Max-pressure (MP) control has emerged as a prominent real-time network traffic signal control strategy due to its simplicity, decentralized structure, and theoretical guarantees of network queue stability. Meanwhile, advances in connected vehicle (CV) technology have sparked extensive research into CV-based traffic signal control. Despite these developments, few studies have investigated MP control in heterogeneously distributed and partially CV environments while ensuring network queue stability. To address these research gaps, we propose a CV-based MP control (CV-MP) method that leverages real-time CV travel time information to compute the pressure, thereby incorporating both the spatial distribution and temporal delays of vehicles, unlike existing approaches that utilized only spatial distribution or temporal delays. In particular, we establish sufficient conditions for road network queue stability that are compatible with most existing MP control methods. Moreover, we pioneered the proof of network queue stability even if the vehicles are only partially connected and heterogeneously distributed, and gave a necessary condition of CV observation for maintaining the stability. Evaluation results on an Amsterdam corridor show that CV-MP significantly reduces vehicle delays compared to both actuated control and conventional MP control across various CV penetration rates. Moreover, in scenarios with dynamic traffic demand, CV-MP achieves lower spillover peaks even with low and heterogeneous CV penetration rates, further highlighting its effectiveness and robustness.

Learning-based control, in particularReinforcement Learning (RL) reinforcementReinforcement learning, and optimization-based control, in particular model predictive control, each have their advantages and disadvantages for online, real-timeOptimal control optimal controlOptimal control of systems with complex dynamicsDynamic. However, both approaches are highly complementary and therefore there is an increased interest in combining their advantages in an integrated approach. In this chapter, we provide an overview of recent results, challenges, and opportunities on an integrated learning-based and optimization-based control approach. We focus in particular on piecewise affine systems as they are an extension of linear systemsLinear systems that can model or approximate hybridHybrid or nonlinearNonlinearbehaviorBehavior and as they still allow for effective numerical solutionSolution approaches.

The railway timetable rescheduling problem is regarded as an efficient way to handle disturbances. Typically, it is tackled using a mixed integer linear programming (MILP) formulation. In this paper, an algorithm that combines both reinforcement learning and optimization is proposed to solve the railway timetable rescheduling problem. Specifically, a value-based reinforcement learning algorithm is implemented to determine the independent integer variables of the MILP problem. Then, the values of all the integer variables can be derived from these independent integer variables. With the solution for the integer variables, the MILP problem can be transformed into a linear programming problem, which can be solved efficiently. The simulation results show that the proposed method can reduce passenger delays compared with the baseline, while also reducing the solution time.

Parameterized model predictive control (PMPC) is one of the many approaches that have been developed to alleviate the high computational requirement of model predictive control (MPC), and it has been shown to significantly reduce the computational complexity while providing comparable control performance with conventional MPC. However, PMPC methods still require a sufficiently accurate model to guarantee the control performance. To deal with model mismatches caused by the changing environment and by disturbances, this paper first proposes a novel framework that uses reinforcement learning (RL) to adapt all components of the PMPC scheme in an online way. More specifically, the novel framework integrates various strategies to adjust different components of PMPC (e.g., objective function, state-feedback control function, optimization settings, and system model), which results in a synthesis framework for RL-based adaptive PMPC. We show that existing adaptive (P)MPC approaches can also be embedded in this synthesis framework. The resulting combined RL-PMPC framework provides a solution for an efficient MPC approach that can deal with model mismatches. A case study is performed in which the framework is applied to freeway traffic control. Simulation results show that for the given case study the RL-based adaptive PMPC approach reduces computational complexity by 98% on average compared to conventional MPC while achieving better control performance than the other controllers, in the presence of model mismatches and disturbances.

While Model Predictive Control (MPC) is a promising approach for network-wide control of urban traffic, the computational complexity of the, often nonlinear, online optimization procedure is too high for real-time implementations. In order to make MPC computationally efficient, this paper introduces a parameterized MPC (PMPC) approach for urban traffic networks that uses Grammatical Evolution to construct continuous parameterized control laws using an effective simulation-based training framework. Furthermore, a projection-based method is proposed to remove the nonlinear constraints that are imposed on the parameters of the parameterized control laws and to guarantee the feasibility of the solution of the MPC optimization problem. The performance and computational efficiency of the constructed parameterized control laws are compared to those of a conventional MPC controller in an extensive simulation-based case study. The results show that the parameterized control laws, which are automatically constructed using Grammatical Evolution, decrease the computational complexity of the online optimization problem by more than 80% with a decrease in performance by less than 10%.

In general, the performance of model-based controllers cannot be guaranteed under model uncertainties or disturbances, while learning-based controllers require an extensively sufficient training process to perform well. These issues especially hold for large-scale nonlinear systems such as urban traffic networks. In this paper, a new framework is proposed by combining model predictive control (MPC) and reinforcement learning (RL) to provide desired performance for urban traffic networks even during the learning process, despite model uncertainties and disturbances. MPC and RL complement each other very well, since MPC provides a sub-optimal and constraint-satisfying control input while RL provides adaptive control laws and can handle uncertainties and disturbances. The resulting combined framework is applied for traffic signal control (TSC) of an urban traffic network. A case study is carried out to compare the performance of the proposed framework and other baseline controllers. Results show that the proposed combined framework outperforms conventional control methods under system uncertainties, in terms of reducing traffic congestion.