Incorporating vision-based artificial intelligence and large language model for smart traffic light control

Abstract

The increasingly complicated urban traffic patterns lead traffic signal control to a new trend of higher flexibility and quicker response, which becomes possible with advances in both sensor technology and artificial intelligence. Though in its early stage, existing intelligent signal controllers equipped with reinforcement learning (RL)-based feature extractor and large language model (LLM)-driven scenario understanding and decision support already demonstrate powerful data digesting ability. This study thus proposes a smart traffic light control system integrating a vision-based perception tool to extract traffic state from real-time snapshot image of the intersection, and an LLM agent controller for signal phase switching upon scenario analysis. An indicator describing the urgency for green time at phase level is defined to abstract the contextual information regarding the competition of multiple approaching traffic flows, which augments the LLM with domain-specific logical reasoning for signal control action generation, aimed at assigning green time to the flows with the most compelling needs. With a RL-based controller providing initial control decision as backup, the proposed method is able to handle both pre-trained and out-of-distribution scenarios through real-time traffic state diagnosis and knowledgeable reasoning. Simulation evaluation on different intersection layouts and vehicle compositions is conducted with horizontal comparison of five benchmarks. A decrease in average waiting time was realized by more than 5 % under normal traffic scenario and 20 % under emergency vehicle scenario, respectively. Further, comprehensive analysis was conducted to explore the applicability of the proposed method and feasibility for real-world application in unmanned aerial vehicle (UAV)-based intelligent traffic management.