Online Planning in POMDPs with Self-Improving Simulators
Jinke He (TU Delft - Electrical Engineering, Mathematics and Computer Science)
Miguel Suau (TU Delft - Electrical Engineering, Mathematics and Computer Science)
Hendrik Baier (Centrum Wiskunde & Informatica (CWI))
Michael Kaisers (Centrum Wiskunde & Informatica (CWI))
Frans A. Oliehoek (TU Delft - Electrical Engineering, Mathematics and Computer Science)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
How can we plan efficiently in a large and complex environment when the time budget is limited? Given the original simulator of the environment, which may be computationally very demanding, we propose to learn online an approximate but much faster simulator that improves over time. To plan reliably and efficiently while the approximate simulator is learning, we develop a method that adaptively decides which simulator to use for every simulation, based on a statistic that measures the accuracy of the approximate simulator. This allows us to use the approximate simulator to replace the original simulator for faster simulations when it is accurate enough under the current context, thus trading off simulation speed and accuracy. Experimental results in two large domains show that when integrated with POMCP, our approach allows to plan with improving efficiency over time.