Plannable Approximations to MDP Homomorphisms: Equivariance under Actions

Plannable Approximations to MDP Homomorphisms: Equivariance under Actions

van der Pol, Elise (Universiteit van Amsterdam)
Kipf, Thomas (Universiteit van Amsterdam)
Oliehoek, F.A. (TU Delft Interactive Intelligence)
Welling, Max (Universiteit van Amsterdam)

An, Bo (editor)
El Fallah Seghrouchni, Amal (editor)
Sukthankar, Gita (editor)

2020-05-09

This work exploits action equivariance for representation learning in reinforcement learning. Equivariance under actions states that transitions in the input space are mirrored by equivalent transitions in latent space, while the map and transition functions should also commute. We introduce a contrastive loss function that enforces action equivariance on the learned representations. We prove that when our loss is zero, we have a homomorphism of a deterministic Markov Decision Process (MDP). Learning equivariant maps leads to structured latent spaces, allowing us to build a model on which we plan through value iteration. We show experimentally that for deterministic MDPs, the optimal policy in the abstract MDP can be successfully lifted to the original MDP. Moreover, the approach easily adapts to changes in the goal states. Empirically, we show that in such MDPs, we obtain better representations in fewer epochs compared to representation learning approaches using reconstructions, while generalizing better to new goals than model-free approaches.

Equivariance
MDP Homomorphisms
MDPs
Planning

http://resolver.tudelft.nl/uuid:7e6a1c28-5896-4e5d-89bc-e21fe5fc0e21

International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), Richland, SC

9781450375184

Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2020

AAMAS 2020, 2020-05-09 → 2020-05-13, Auckland, New Zealand

Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS, 1548-8403, 2020-May

Institutional Repository

conference paper

© 2020 Elise van der Pol, Thomas Kipf, F.A. Oliehoek, Max Welling