Eligibility traces and forgetting factor in recursive least-squares-based temporal difference

Journal article (2022)

Authors

S. Baldi Team Bart De Schutter - Mechanical, Maritime and Materials Engineering , Southeast University
Z. Zhang Southeast University
Di Liu Southeast University, University Medical Center Groningen
Research Group
Team Bart De Schutter (Mechanical, Maritime and Materials Engineering) (TU Delft)
Copyright: © 2022 S. Baldi, Z. Zhang, Di Liu
DOI: https://doi.org/10.1002/acs.3282
Reinforcement learning Least squares Eligibility traces Instrumental variable method Temporal difference
More Info
expand_more

Published Date

2022

Language

English

Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Faculty

Mechanical, Maritime and Materials Engineering

Department

Delft Center for Systems and Control

Research Group

Team Bart De Schutter

Abstract

We propose a new reinforcement learning method in the framework of Recursive Least Squares-Temporal Difference (RLS-TD). Instead of using the standard mechanism of eligibility traces (resulting in RLS-TD((Formula presented.))), we propose to use the forgetting factor commonly used in gradient-based or least-square estimation, and we show that it has a similar role as eligibility traces. An instrumental variable perspective is adopted to formulate the new algorithm, referred to as RLS-TD with forgetting factor (RLS-TD-f). An interesting aspect of the proposed algorithm is that it has an interpretation of a minimizer of an appropriate cost function. We test the effectiveness of the algorithm in a Policy Iteration setting, meaning that we aim to improve the performance of an initially stabilizing control policy (over large portion of the state space). We take a cart-pole benchmark and an adaptive cruise control benchmark as experimental platforms.