Multi-Agent Decision-Making Modes in Uncertain Interactive Traffic Scenarios via Graph Convolution-Based Deep Reinforcement Learning

Multi-Agent Decision-Making Modes in Uncertain Interactive Traffic Scenarios via Graph Convolution-Based Deep Reinforcement Learning

Gao, Xin (Beijing Institute of Technology)
Li, Xueyuan (Beijing Institute of Technology)
Liu, Qi (Beijing Institute of Technology)
Li, Z. (TU Delft Transport and Planning; Beijing Institute of Technology)
Yang, Fan (Beijing Institute of Technology)
Luan, Tian (Beijing Institute of Technology)

2022

As one of the main elements of reinforcement learning, the design of the reward function is often not given enough attention when reinforcement learning is used in concrete applications, which leads to unsatisfactory performances. In this study, a reward function matrix is proposed for training various decision-making modes with emphasis on decision-making styles and further emphasis on incentives and punishments. Additionally, we model a traffic scene via graph model to better represent the interaction between vehicles, and adopt the graph convolutional network (GCN) to extract the features of the graph structure to help the connected autonomous vehicles perform decision-making directly. Furthermore, we combine GCN with deep Q-learning and multi-step double deep Q-learning to train four decision-making modes, which are named the graph convolutional deep Q-network (GQN) and the multi-step double graph convolutional deep Q-network (MDGQN). In the simulation, the superiority of the reward function matrix is proved by comparing it with the baseline, and evaluation metrics are proposed to verify the performance differences among decision-making modes. Results show that the trained decision-making modes can satisfy various driving requirements, including task completion rate, safety requirements, comfort level, and completion efficiency, by adjusting the weight values in the reward function matrix. Finally, the decision-making modes trained by MDGQN had better performance in an uncertain highway exit scene than those trained by GQN.

connected autonomous vehicles
GQN
MDGQN
multi-mode decision-making
reward function matrix
uncertain highway exit scene

http://resolver.tudelft.nl/uuid:6302ab05-4e3f-4952-87e9-2804fdbe9f01

https://doi.org/10.3390/s22124586

1424-8220

Sensors, 22 (12)

Institutional Repository

journal article

© 2022 Xin Gao, Xueyuan Li, Qi Liu, Z. Li, Fan Yang, Tian Luan