Learning state representation for deep actor-critic control

Learning state representation for deep actor-critic control

Munk, J.
Kober, J. (TU Delft OLD Intelligent Control & Robotics)
Babuska, R. (TU Delft OLD Intelligent Control & Robotics)

Bullo, Francesco (editor)
Prieur, Christophe (editor)
Giua, Alessandro (editor)

2016

Deep Neural Networks (DNNs) can be used as function approximators in Reinforcement Learning (RL). One advantage of DNNs is that they can cope with large input dimensions. Instead of relying on feature engineering to lower the input dimension, DNNs can extract the features from raw observations. The drawback of this end-to-end learning is that it usually requires a large amount of data, which for real-world control applications is not always available. In this paper, a new algorithm, Model Learning Deep Deterministic Policy Gradient (ML-DDPG), is proposed that combines RL with state representation learning, i.e., learning a mapping from an input vector to a state before solving the RL task. The ML-DDPG algorithm uses a concept we call predictive priors to learn a model network which is subsequently used to pre-train the first layer of the actor and critic networks. Simulation results show that the ML-DDPG can learn reasonable continuous control policies from high-dimensional observations that contain also task-irrelevant information. Furthermore, in some cases, this approach significantly improves the final performance in comparison to end-to-end learning.

Approximation algorithms
Robot sensing systems
Algorithm design and analysis
Prediction algorithms
Learning (artificial intelligence)
Feature extraction

http://resolver.tudelft.nl/uuid:1830de68-f008-471f-898f-0665c2a907d2

https://doi.org/10.1109/CDC.2016.7798980

IEEE, Piscataway, NJ, USA

978-1-5090-1837-6

Proceedings 2016 IEEE 55th Conference on Decision and Control (CDC)

55th IEEE Conference on Decision and Control, CDC 2016, 2016-12-12 → 2016-12-14, Las Vegas, United States

Accepted Author Manuscript

Institutional Repository

conference paper

© 2016 J. Munk, J. Kober, R. Babuska