Optimal control via reinforcement learning with symbolic policy approximation

Kubalík, Jiří; Alibekov, Eduard; Babuska, R.

doi:10.1016/j.ifacol.2017.08.805

Optimal control via reinforcement learning with symbolic policy approximation

Title

Optimal control via reinforcement learning with symbolic policy approximation

Author

Kubalík, Jiří (Czech Technical University)
Alibekov, Eduard (Czech Technical University)
Babuska, R. (TU Delft Learning & Autonomous Control)

Date

2017

Abstract

Model-based reinforcement learning (RL) algorithms can be used to derive optimal control laws for nonlinear dynamic systems. With continuous-valued state and input variables, RL algorithms have to rely on function approximators to represent the value function and policy mappings. This paper addresses the problem of finding a smooth policy based on the value function represented by means of a basis-function approximator. We first show that policies derived directly from the value function or represented explicitly by the same type of approximator lead to inferior control performance, manifested by non-smooth control signals and steady-state errors. We then propose a novel method to construct a smooth policy represented by an analytic equation, obtained by means of symbolic regression. The proposed method is illustrated on a reference-tracking problem of a 1-DOF robot arm operating under the influence of gravity. The results show that the analytic control law performs at least equally well as the original numerically approximated policy, while it leads to much smoother control signals. In addition, the analytic function is readable (as opposed to black-box approximators) and can be used in further analysis and synthesis of the closed loop.

Subject

genetic programming
nonlinear model-based control
optimal control
reinforcement learning
symbolic regression
value iteration

To reference this document use:

http://resolver.tudelft.nl/uuid:5679b1a1-9a8a-42e4-b8d8-012296cea0a0

DOI

https://doi.org/10.1016/j.ifacol.2017.08.805

Source

IFAC-PapersOnLine, 50 (1), 4162-4167

Event

20th World Congress of the International Federation of Automatic Control (IFAC), 2017, 2017-07-09 → 2017-07-14, Toulouse, France

Part of collection

Institutional Repository