CEM: Constrained Entropy Maximization for Task-Agnostic Safe Exploration

CEM: Constrained Entropy Maximization for Task-Agnostic Safe Exploration

Yang, Q. (TU Delft Algorithmics)
Spaan, M.T.J. (TU Delft Algorithmics)

2023

Without an assigned task, a suitable intrinsic objective for an agent is to explore the environment efficiently. However, the pursuit of exploration will inevitably bring more safety risks.
An under-explored aspect of reinforcement learning is how to achieve safe efficient exploration when the task is unknown.
In this paper, we propose a practical Constrained Entropy Maximization (CEM) algorithm to solve task-agnostic safe exploration problems, which naturally require a finite horizon and undiscounted constraints on safety costs.
The CEM algorithm aims to learn a policy that maximizes the state entropy under the premise of safety.
To avoid approximating the state density in complex domains, CEM leverages a $k$-nearest neighbor entropy estimator to evaluate the efficiency of exploration.
In terms of safety, CEM minimizes the safety costs, and adaptively trades off safety and exploration based on the current constraint satisfaction. We empirically show that CEM allows learning a safe exploration policy in complex continuous-control domains, and the learned policy benefits downstream tasks in safety and sample efficiency.

Reinforcement Learning
Safe Exploration

http://resolver.tudelft.nl/uuid:e473c39b-e977-44fe-bf01-d94292f713db

2023-10-05

The Thirty-Seventh AAAI Conference on Artificial Intelligence (AAAI-23)

37th AAAI Conference on Artificial Intelligence, 2023-02-07 → 2023-02-14, Washington, United States

Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

Institutional Repository

conference paper

© 2023 Q. Yang, M.T.J. Spaan