Using Bisimulation Metrics to Analyze and Evaluate Latent State Representations

Using Bisimulation Metrics to Analyze and Evaluate Latent State Representations

Albers, N. (TU Delft Interactive Intelligence)
Suau, M. (TU Delft Interactive Intelligence)
Oliehoek, F.A. (TU Delft Interactive Intelligence)

Leiva, Edit Luis A. (editor)
Pruski, Cédric (editor)
Markovich, Réka (editor)
Najjar, Amro (editor)
Schommer, Christoph (editor)

2021

Deep Reinforcement Learning (RL) is a promising technique towards constructing intelligent agents, but it is not always easy to understand the learning process and the factors that impact it. To shed some light on this, we analyze the Latent State Representations (LSRs) that deep RL agents learn, and compare them to what such agents should ideally learn. We propose a crisp definition of ’ideal LSR’ based on a bisimulation metric, which measures how behaviorally similar states are. The ideal LSR is that in which the distance between two states is proportional to this bisimulation metric. Intuitively, forming such an ideal representation is highly favorable due to its compactness and generalization properties. Here we investigate if this type of representation is also desirable in practice. Our experiments suggest that learning representations that are close to this ideal LSR may improve upon generalization to new irrelevant feature values and modified dynamics. Yet, we show empirically that the extent to which such representations are learned depends on both the network capacity and the state encoding, and that with the current techniques the exact ideal LSR is never formed.

Deep Reinforcement Learning
Bisimulation Metrics

http://resolver.tudelft.nl/uuid:cdb0d337-dd76-43f2-8f3d-d5a005af0790

BNAIC/BeneLearn 2021: 33rd Benelux Conference on Artificial Intelligence and 30th Belgian-Dutch Conference on Machine Learning

33rd Benelux Conference on Artificial Intelligence and
30th Belgian-Dutch Conference on Machine Learning, 2021-11-10 → 2021-11-12, Esch-sur-Alzette, Luxembourg

Institutional Repository

conference paper

© 2021 N. Albers, M. Suau, F.A. Oliehoek