Think Too Fast Nor Too Slow

The Computational Trade-off Between Planning And Reinforcement Learning

Book chapter (2020)

Authors

T.M. Moerland Universiteit Leiden, Interactive Intelligence -

Anna Deichler Student

S. Baldi Team Bart De Schutter - Mechanical, Maritime and Materials Engineering , Southeast University

D.J. Broekens Universiteit Leiden

C.M. Jonker Universiteit Leiden, Interactive Intelligence -

Research Group

Interactive Intelligence () (TU Delft)

More Info

expand_more

To reference this document use:

http://resolver.tudelft.nl/uuid:f46e64c0-a1ce-457e-bf56-748d3bb73673

Published Date

2020

Language

English

Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Faculty

Electrical Engineering, Mathematics and Computer Science

Department

Intelligent Systems

Research Group

Interactive Intelligence

Abstract

Planning and reinforcement learning are two key approaches to sequential decision making. Multi-step approximate real-time dynamic programming, a recently successful algorithm class of which AlphaZero [Silver et al., 2018] is an example, combines both by nesting planning within a learning loop. However, the combination of planning and learning introduces a new question: how should we balance time spend on planning, learning and acting? The importance of this trade-off has not been explicitly studied before. We show that it is actually of key importance, with computational results indicating that we should neither plan too long nor too short. Conceptually, we identify a new spectrum of planning-learning algorithms which ranges from exhaustive search (long planning) to model-free RL (no planning), with optimal performance achieved midway.

Files

2005.07404_1.pdf

(.pdf | 2.69 Mb)