Generalisation Ability of Proper Value Equivalence Models in Model-Based Reinforcement Learning

Bachelor thesis (2024)

Authors

S. Bratus Electrical Engineering, Mathematics and Computer Science

Contributors

J. He Sequential Decision Making - EEMCS (mentor)
M.M. de Weerdt Algorithmics - EEMCS (coach)
F.A. Oliehoek Sequential Decision Making - EEMCS (graduation committee member)
Faculty
Electrical Engineering, Mathematics and Computer Science
Generalisation Model-Based Reinforcement Learning Value Equivalence
More Info
expand_more

Published Date

25-06-2024

Language

English

Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Faculty

Electrical Engineering, Mathematics and Computer Science

Abstract

We investigate the generalization performance of predictive models in model-based reinforcement learning when trained using maximum likelihood estimation (MLE) versus proper value equivalence (PVE) loss functions. While the more conventional MLE loss aims to fit models to predict state transitions and rewards as accurately as possible, value-equivalent methods (e.g. PVE) prioritize value-relevant features. We show that in a tabular setting, MLE-based models generalize better than their PVE counterparts when fit to a small number of training policies, whereas PVE-based models perform better as the number of policies increases. With increasing model rank, generalisation error tends to improve for MLE and PVE, and the two become closer in generalisation ability.