The algorithm automatically customizes reward functions based on the classification of traffic regimes to solve nonstationary traffic signal control problems 

Existing traffic signal controllers are often inadequate in handling traffic congestion and reinforcement learning (RL) has been adopted to solve this problem. However, RL usually works in a stationary environment, while traffic environments are nonstationary. Researchers have now developed a meta-RL model that adjusts its goal based on the traffic environment, maximizing vehicle throughput during peak hours and minimizing delays during relaxed traffic flow. The algorithm has a wide-area coverage and outperforms existing alternative algorithms.

 Traffic signal control affects the daily life of people living in urban areas. The existing system relies on a theory- or rule-based controller in charge of altering the traffic lights based on traffic conditions. The objective is to reduce vehicle delay during unsaturated traffic conditions and maximize the vehicle throughput during congestion. However, the existing traffic signal controller cannot fulfil such altering objectives, and a human controller can only manage a few intersections. In view of this, recent advancements in artificial intelligence have focused towards enabling alternate ways of traffic signal control.

 Current research on this front has explored reinforcement learning (RL) algorithms as a possible approach. However, RL algorithms do not always work due to the dynamic nature of traffic environments, i.e., traffic at an intersection depends on traffic conditions at other nearby junctions. While multiagent RL can tackle this interference issue, it suffers from exponentially growing dimensionality with the increase in intersections.

 Recently, a team of researchers from Chung Ang University in Korea led by Prof. Keemin Sohn proposed a meta-RL model to solve this issue. Specifically, the team developed an extended deep Q-network (EDQN)-incorporated context-based meta-RL model for traffic signal control. “Existing studies have devised meta-RL algorithms based on intersection geometry, traffic signal phases, or traffic conditions. The present research deals with the non-stationary aspect of signal control according to the congestion levels. The meta-RL works autonomously in detecting traffic states, classifying traffic regimes, and assigning signal phases,” explains Prof. Sohn, speaking of their study, which was made available online on 30 September 2022 and was published in the Computer-Aided Civil and Infrastructure Engineering journal on 30 September 2022.

 The model works as follows. It determines the traffic regime–saturated or unsaturated–by utilizing a latent variable that indicates the overall environmental condition. Based on traffic flow, the model either maximizes throughput or minimizes delays similar to a human controller. It does so by implementing traffic signal phases (action). As with intelligent learning agents, the action is controlled by the provision of a “reward.” Here, the reward function is set to be +1 or -1 corresponding to a better or worse performance in handling traffic relative to the previous interval, respectively. Further, the EDQN acts as a decoder to jointly control traffic signals for multiple intersections.

 Following its theoretical development, the researchers trained and tested their meta-RL algorithm using Vissim v21.0, a commercial traffic simulator, to mimic real-world traffic conditions. Further, a transportation network in southwest Seoul consisting of 15 intersections was chosen as a real-world testbed. Following meta-training, the model could adapt to new tasks during meta-testing without adjusting its parameters.

 The simulation experiments revealed that the proposed model could switch control tasks (via transitions) without any explicit traffic information. It could also differentiate between rewards according to the saturation level of traffic conditions. Further, the EDQN-based meta-RL model outperformed the existing algorithms for traffic signal control and could be extended to tasks with different transitions and rewards.

 Nevertheless, the researchers pointed to the need for an even more precise algorithm to consider different saturation levels from intersection to intersection. “Existing research has employed reinforcement learning for traffic signal control with a single fixed objective. In contrast, this work has devised a controller that can autonomously select the optimal target based on the latest traffic condition. The framework, if adopted by traffic signal control agencies, could yield travel benefits that have never been experienced before,” concludes an optimistic Prof. Sohn.

 Reference

About Chung-Ang University

Chung-Ang University is a private comprehensive research university located in Seoul, South Korea. It was started as a kindergarten in 1916 and attained university status in 1953. It is fully accredited by the Ministry of Education of Korea. Chung-Ang University conducts research activities under the slogan of “Justice and Truth.” Its new vision is “The Global Creative Leader” upon completion of 100 years. Chung-Ang University offers undergraduate, postgraduate, and doctoral programs that encompass a law school, management program, and medical school; it has 16 undergraduate and graduate schools each. Chung-Ang University’s culture and arts programs are considered the best in Korea.

Website: https://neweng.cau.ac.kr/index.do

About Professor Keemin Sohn

Keemin Sohn is a Professor in the school of Civil and Environmental Engineering at Chung-Ang University, Korea. He leads the laboratory of big data applications in the public sector. His research interests encompass data science and artificial intelligence with applications in transportation planning, operation, and management. He has published more than 50 papers in rigorous peer-reviewed journals, including Computer-Aided Civil and Infrastructure Engineering, Transportation Research (Part A, B, and C), Transportation Science, IEEE Transactions on Intelligent Transport Systems, and IEEE Transactions on Vehicular Technology.