CA

Christopher Amato

info

Please Note

5 records found

Bayes-Adaptive Deep Dropout RL for POMDPs

Conference paper (2022) - Sammie Katt, Hai Nguyen, Frans A. Oliehoek, Christopher Amato
While reinforcement learning (RL) has made great advances in scalability, exploration and partial observability are still active research topics. In contrast, Bayesian RL (BRL) provides a principled answer to both state estimation and the exploration-exploitation trade-off, but struggles to scale. To tackle this challenge, BRL frameworks with various prior assumptions have been proposed, with varied success. This work presents a representation-agnostic formulation of BRL under partially observability, unifying the previous models under one theoretical umbrella. To demonstrate its practical significance we also propose a novel derivation, Bayes-Adaptive Deep Dropout rl (BADDr), based on dropout networks. Under this parameterization, in contrast to previous work, the belief over the state and dynamics is a more scalable inference problem. We choose actions through Monte-Carlo tree search and empirically show that our method is competitive with state-of-the-art BRL methods on small domains while being able to solve much larger ones. ...
Conference paper (2019) - Sammie Katt, Frans A. Oliehoek, Christopher Amato
Model-based Bayesian Reinforcement Learning (BRL) provides a principled solution to dealing with the exploration-exploitation trade-off, but such methods typically assume a fully observable environments. The few Bayesian RL methods that are applicable in partially observable domains, such as the Bayes-Adaptive POMDP (BA-POMDP), scale poorly. To address this issue, we introduce the Factored BA-POMDP model (FBA-POMDP), a framework that is able to learn a compact model of the dynamics by exploiting the underlying structure of a POMDP. The FBA-POMDP framework casts the problem as a planning task, for which we adapt the Monte-Carlo Tree Search planning algorithm and develop a belief tracking method to approximate the joint posterior over the state and model variables. Our empirical results show that this method outperforms a number of BRL baselines and is able to learn efficiently when the factorization is known, as well as learn both the factorization and the model parameters simultaneously. ...
Conference paper (2018) - Sammie Katt, Frans A. Oliehoek, Christopher Amato
While the POMDP has proven to be a powerful framework to model and solve partially observable stochastic problems, it assumes ac- curate and complete knowledge of the environment. When such information is not available, as is the case in many real world appli- cations, one must learn such a model. The BA-POMDP considers the model as part of the hidden state and explicitly considers the uncertainty over it, and as a result transforms the learning problem into a planning problem. This model, however, grows exponentially with the underlying POMDP size, and becomes intractable for non- trivial problems. In this article we propose a factored framework, the FBA-POMDP that represents the model as a Bayes-Net, dras- tically decreasing the number of parameters required to describe the dynamics of the environment. We demonstrate that the our ap- proach allows solvers to tackle problems much larger than possible in the BA-POMDP. ...
Journal article (2018) - Christopher Amato, Haitham Bou Ammar, More Authors..., Elizabeth Churchill, Erez Karpas, Takashi Kido, Mike Kuniavsky, W. F. Lawless, Frans Oliehoek, Francesca Rossi, Stephen Russell
The Association for the Advancement of Artifcial Intelligence, in cooperation with Stanford University's Department of Computer Science, presented the 2018 Spring Symposium Series, held March 26-28, 2018, on the campus of Stanford University. The seven symposia held were AI and Society: Ethics, Safety, and Trustworthiness in Intelligent Agents; Artifcial Intelligence for the Internet of Everything; Beyond Machine Intelligence: Understanding Cognitive Bias and Humanity for WellBeing AI; Data-Effcient Reinforcement Learning; The Design of the User Experience for Artifcial Intelligence (the UX of AI); Integrated Representation, Reasoning, Learning, and Execution for Goal-Directed Autonomy; Learning, Inference, and Control of Multiagent Systems. This report, compiled from organizers of the symposia, summarizes the research of the symposia that took place. ...