Effects of Partial Observability Solver Methods on Training and Final Policies in Autonomous Driver RL

Effects of Partial Observability Solver Methods on Training and Final Policies in Autonomous Driver RL: How do different methods for dealing with partial observability in the environment influence training and the robustness of final policies under various testing conditions?

Çil, Ata (TU Delft Electrical Engineering, Mathematics and Computer Science; TU Delft Algorithmics)

Zanger, M.A. (mentor)
Spaan, M.T.J. (mentor)
Congeduti, E. (graduation committee)

Delft University of Technology

Delft University of Technology

Computer Science and Engineering

CSE3000 Research Project

2023-06-28

Autonomous driving is a complex problem that can potentially be solved using artificial intelligence. The complexity stems from the system's need to understand the surroundings and make appropriate decisions. However, there are various challenges in constructing such a sophisticated system. One of the main challenges is to make the agent learn from the environmental input and since the environment is not fully observable and constantly changing, the agent should be flexible enough to extract important information from the input. To create such an agent this paper focused on the 3 different methods, specifically the application of frame stacking, long short-term memory (LSTM), and a combination of both methods. To analyze these methods 3 Deep Q Network(DQN) based agents are created. These 3 agents have frame stacking, LSTM, and both combined methods to solve the partially observable problem of autonomous driving. Their training performances are analyzed, and the results revealed significantly different trends in the training and evaluation phase.
Especially the experiment resulted in LSTM being a more robust method but had lower performance than DQN with frame stack, which showed a trade-off between these 2 qualities of an agent. The agent with LSTM and frame stack was able to learn faster at the beginning, but as it was unstable, it got a lower return value from the training run at the end. This instability can be a problem in the real world, especially in autonomous driving, where it is very important to have robust implementation.

Reinforcement Learning
Autonomous car
Partial Observability
long short term memory

http://resolver.tudelft.nl/uuid:148bade0-e225-42b7-9eed-dc886afd6b87

Student theses

bachelor thesis

© 2023 Ata Çil