Intelligent signalized intersection management for mixed traffic using Deep Q-Learning

Abstract

Signalized urban intersections are bottlenecks for traffic and cause congestion. To improve traffic signal plans, research efforts have been made to create self-adaptive traffic controllers, i.e. controllers which adapt in real-time to the current traffic demand based on connected vehicle data. Past research on self-adaptive controllers has mostly assumed that all of the vehicles are connected. Yet, at least for the close future, traffic will consist of a mixture of regular vehicles (RVs) and connected vehicles (CVs). Up to date, few studies investigated whether self-adaptive controllers are able to control traffic signals efficiently under CV-penetration rates of less than 100%. Within literature, different types of methods to create self-adaptive traffic signal controllers were found. In this thesis, it was chosen to focus on deep Q-learning models (DQN), which are a type of reinforcement learning model. This thesis aimed to alleviate the research gap by investigating whether deep Q-learning-based traffic signal controllers are able to reduce traffic congestion for mixed traffic scenarios, and what design choices have to be made to build such a controller. To answer this, this thesis conducted a systematic literature review, designed and systematically fine-tuned two DQN agents and evaluated these DQN agents on different traffic situations. The literature review discussed the most important design choices regarding agent design, traffic environments and model evaluation. These results were used to design a vanilla DQN agent and a recurrent DQN agent. Since not all design choices were clear a priori, the impacts of several alternative choices and parameters on agent stability and performance were systematically investigated in experiments in order to fine-tune the agents. After the two agents were fine-tuned, they were evaluated using a microscopic traffic simulator. It was investigated how stable the agents’ performance is, how well the agents can efficiently control traffic signals under different traffic scenarios (low constant, medium constant, high constant and dynamic traffic) and CV-penetration rates (between 10% and 100%), how robust agents are to changes in penetration rates, and how the vanilla and recurrent agents differ. To be able to benchmark the two agents’ performances, they were compared to a traditional fixed-time controller. It was concluded that the designed vanilla DQN agent is unsuitable for mixed traffic control. The recurrent DQN agent on the other hand, performed better than the vanilla agent in terms of stability, performance and robustness, and only the recurrent agent was able to outperform the fixed-time controller for all but the lowest penetration rates. This makes recurrent DQN a promising method for future research on reinforcement learning-based traffic signal control. In brief, this thesis found that reinforcement learning algorithms could potentially be used to control signalized intersections in mixed traffic scenarios such that traffic congestion is reduced. Yet, DQN controllers are not yet mature to be implemented in real-life and future research to improve them is needed.