Learning-based control, in particularReinforcement Learning (RL) reinforcementReinforcement learning, and optimization-based control, in particular model predictive control, each have their advantages and disadvantages for online, real-timeOptimal control optimal controlOptimal control of systems with complex dynamicsDynamic. However, both approaches are highly complementary and therefore there is an increased interest in combining their advantages in an integrated approach. In this chapter, we provide an overview of recent results, challenges, and opportunities on an integrated learning-based and optimization-based control approach. We focus in particular on piecewise affine systems as they are an extension of linear systemsLinear systems that can model or approximate hybridHybrid or nonlinearNonlinearbehaviorBehavior and as they still allow for effective numerical solutionSolution approaches.

Co-optimization of both vehicle speed and gear position via model predictive control (MPC) has been shown to offer benefits for fuel-efficient autonomous driving. However, optimizing both the vehicle’s continuous dynamics and discrete gear positions may be too computationally intensive for a real-time implementation. This work proposes a learning-based MPC scheme to address this issue. A policy is trained to select and fix the gear positions across the prediction horizon of the MPC controller, leaving a significantly simpler continuous optimization problem to be solved online. In simulation, the proposed approach is shown to have a significantly lower computation burden and a comparable performance, with respect to pure MPC-based co-optimization.

Vulnerable road user safety is paramount for increasing shares of active travel modes and introducing automated vehicles. Microscopic traffic simulation is a prevalent method in research and practice with a growing focus on safety and cyclists. Its practical benefits make it an essential tool for developing safe future transportation. We review the methodology of simulation studies and the validation of their microscopic models to evaluate cycling safety assessment in microscopic simulations. We find that current work relies predominantly on the lane-based models of established traffic flow simulation packages that separate longitudinal and lateral dynamics. These models do not sufficiently capture diverse behaviors and conflict causality to predict cycling safety. In contrast, new models with successful calibrations and validations advance simulated interactions towards capturing conflict causality. Of 42 reviewed studies, six calibrate, and three validate models for safety prediction. Other studies disregard calibration and validation, posing a threat of unfounded safety predictions and unsafe design recommendations. We present a methodological framework conceptualizing best practices for reliable assessment. It calls for the identification of safety-relevant behaviors of cyclists and other road users in conflicts. Specialized behavioral models must be developed, calibrated, and validated. The selected safety indicators must enable capturing the expected unsafe events. To create these tools, improved models of cycling behavior must be transferred to established simulation packages. Following the framework, researchers and practitioners can use simulation as a practical and ethical means to assess the cycling safety impact of innovations ranging from infrastructure to automation and connectivity.

In the backdrop of an increasingly pressing need for effective urban and highway transportation systems, this work explores the synergy between model-based and learning-based strategies to enhance traffic flow management by use of an innovative approach to the problem of ramp metering control that embeds Reinforcement Learning (RL) techniques within the Model Predictive Control (MPC) framework. The control problem is formulated as an RL task by crafting a suitable stage cost function that is representative of the traffic conditions, variability in the control action, and violations of the constraint on the maximum number of vehicles in queue. An MPC-based RL approach, which leverages the MPC optimal problem as a function approximation for the RL algorithm, is proposed to learn to efficiently control an on-ramp and satisfy its constraints despite uncertainties in the system model and variable demands. Simulations are performed on a benchmark small-scale highway network to compare the proposed methodology against other state-of-the-art control approaches. Results show that, starting from an MPC controller that has an imprecise model and is poorly tuned, the proposed methodology is able to effectively learn to improve the control policy such that congestion in the network is reduced and constraints are satisfied, yielding an improved performance that is superior to the other controllers.

Uncertainty in the behavior of other traffic participants is a crucial factor in collision avoidance for automated driving; here, stochastic metrics could avoid overly conservative decisions. This article introduces a stochastic model predictive control (SMPC) planner for emergency collision avoidance in highway scenarios to proactively minimize collision risk while ensuring safety through chance constraints. To guarantee that the emergency trajectory can be attained, we incorporate nonlinear tire dynamics in the prediction model of the ego vehicle. Further, we exploit max-min-plus-scaling (MMPS) approximations of the nonlinearities to avoid conservatism, enforce proactive collision avoidance, and improve computational efficiency in terms of performance and speed. Consequently, our contributions include integrating a dynamic ego vehicle model into the SMPC planner, introducing the MMPS approximation for real-time implementation in emergency scenarios, and integrating SMPC with hybridized chance constraints and risk minimization. We evaluate our SMPC formulation in terms of proactivity and efficiency in various hazardous scenarios. Moreover, we demonstrate the effectiveness of our proposed approach by comparing it with a state-of-the-art SMPC planner and we validate that the generated trajectories can be attained using a high-fidelity vehicle model in IPG CarMaker.

Greenhouse climate control is concerned with maximizing performance in terms of crop yield and resource efficiency. One promising approach is model predictive control (MPC), which leverages a model of the system to optimize the control inputs, while enforcing physical constraints. However, prediction models for greenhouse systems are inherently inaccurate due to the complexity of the real system and the uncertainty in predicted weather profiles. For model-based control approaches such as MPC, this can degrade performance and lead to constraint violations. Existing approaches address uncertainty in the prediction model with robust or stochastic MPC methodology; however, these necessarily reduce crop yield due to conservatism and often bear higher computational loads. In contrast, learning-based control approaches, such as reinforcement learning (RL), can handle uncertainty naturally by leveraging data to improve performance. This work proposes an MPC-based RL control framework to optimize the climate control performance in the presence of prediction uncertainty. The approach employs a parametrized MPC scheme that learns directly from data, in an online fashion, the parametrization of the constraints, prediction model, and optimization cost that minimizes constraint violations and maximizes climate control performance. Simulations show that the approach can learn an MPC controller that significantly outperforms the current state-of-the-art in terms of constraint violations and efficient crop growth.

This paper presents a novel approach for distributed model predictive control (MPC) for piecewise affine (PWA) systems. Existing approaches rely on solving mixed-integer optimization problems, requiring significant computation power or time. We propose a distributed MPC scheme that requires solving only convex optimization problems. The key contribution is a novel method, based on the alternating direction method of multipliers, for solving the non-convex optimal control problem that arises due to the PWA dynamics. We present a distributed MPC scheme, leveraging this method, that explicitly accounts for the coupling between subsystems by reaching agreement on the values of coupled states. Stability and recursive feasibility are shown under additional assumptions on the underlying system. Two numerical examples are provided, in which the proposed controller is shown to significantly improve the CPU time and closed-loop performance over existing state-of-the-art approaches.

The railway timetable rescheduling problem is regarded as an efficient way to handle disturbances. Typically, it is tackled using a mixed integer linear programming (MILP) formulation. In this paper, an algorithm that combines both reinforcement learning and optimization is proposed to solve the railway timetable rescheduling problem. Specifically, a value-based reinforcement learning algorithm is implemented to determine the independent integer variables of the MILP problem. Then, the values of all the integer variables can be derived from these independent integer variables. With the solution for the integer variables, the MILP problem can be transformed into a linear programming problem, which can be solved efficiently. The simulation results show that the proposed method can reduce passenger delays compared with the baseline, while also reducing the solution time.

Virtual coupling is regarded as an efficient way to improve the line capacity of rail transportation systems by reducing the spacing between consecutive trains. This paper is the first to compare and assess different distributed model predictive control (MPC) approaches, i.e., cooperative distributed MPC, serial distributed MPC, and decentralized MPC, for virtually coupled trains with a nonlinear train dynamic model. To make a balanced trade-off between computational complexity and efficiency, we also propose and assess convex approximations of the above control approaches. Furthermore, we are the first to introduce the relaxed dynamic programming approach to analyze the stability of the MPC-based nonlinear train control problem. By using the relaxed dynamic programming approach, a distributed stopping criterion with a stability guarantee is developed for the cooperative distributed MPC approach. In real life, masses of trains are different and can change at stations due to changes in passenger loads. This change in mass can significantly affect the dynamics and control of the virtually coupled trains when not taken into account in the control design. Therefore, we explicitly consider heterogeneous train masses when designing MPC approaches. We evaluate the different distributed MPC approaches through case studies based on the data of the Beijing Yizhuang Line. Simulation results indicate that the cooperative distributed MPC approach has the best tracking performance, while the serial distributed MPC approach can reduce communication requirements and computation capabilities with sacrifices of tracking performance.

Microscopic simulation is an established tool in traffic engineering and research, where aggregated traffic performance measures are inferred from the simulation of individual agents. Additionally, measures describing the safety and efficiency of road user interactions gain importance for recent developments such as automated vehicles and urban cycling. However, current simulation frameworks model interactions including cyclists only with limited realism. To address this issue, we propose to bring bicycle dynamics to traffic simulation. We demonstrate that a novel reformulation of the social force framework can create input signals for a controlled inverted pendulum bicycle model and thereby enable a fully two-dimensional open space simulation of cyclist interactions. The inverted pendulum model introduces the need to stabilize the bicycle as a constraint to the reactive behavior of simulated cyclists. Furthermore, it enables the simulation of countersteering and weaving for stabilization. Our cyclist social forces have anisotropic force fields with respect to relative interaction position and orientation to describe the varying interaction constellations in open space. With these models, we simulate five single- and multi-cyclist test cases and show that the generated trajectories notably differ from results obtained from a 2D bicycle model without lean angle simulation. Measurements of the maximum lateral path deviation and post-encroachment time show that these differences are relevant for typical applications. Our work demonstrates the potential of introducing physics-based realistic bicycle dynamics to the microscopic simulation of individual road user interactions and the fundamental capability of our reformulated cyclist social forces to do so. Going further, we plan to calibrate and validate our model based on naturalistic cycling data to support the initial results of this work.

This paper presents a novel approach to multi-agent reinforcement learning (RL) for linear systems with convex polytopic constraints. Existing work on RL has demonstrated the use of model predictive control (MPC) as a function approximator for the policy and value functions. The current paper is the first work to extend this idea to the multi-agent setting. We propose the use of a distributed MPC scheme as a function approximator, with a structure allowing for distributed learning and deployment. We then show that Q-learning updates can be performed distributively without introducing nonstationarity, by reconstructing a centralized learning update. The effectiveness of the approach is demonstrated on a numerical example.

The majority of computer vision architectures are developed based on the assumption of the availability of good quality data. However, this is a particularly hard requirement to achieve in underwater conditions. To address this limitation, plenty of underwater image enhancement methods have received considerable attention during the last decades, but due to the lack of a commonly accepted framework to systematically evaluate them and to determine the likely optimal one for a given image, their adoption in practice is hindered, since it is not clear which one can achieve the best results. In this paper, we propose a standardized selection framework to evaluate the quality of an underwater image and to estimate the most suitable image enhancement technique based on its impact on the image classification performance.

This paper deals with the train scheduling problem for metro networks taking into account time-dependent passenger origin–destination demands and train speed profiles. The aim is to adjust train schedules online according to time-dependent passenger demands so that passenger satisfaction and operational costs are jointly optimized. An extended passenger absorption model that explicitly includes time-dependent passenger origin–destination demands is developed, where the term “absorption” refers to passengers boarding trains. Then, the passenger absorption model is extended to a bi-level framework, where passenger demands and rolling stock availability are considered at the higher level, and detailed timetables and train speed profiles are included at the lower level. A bi-level model predictive control (MPC) approach is developed for the integrated problem. The optimization problems of both levels of the bi-level MPC approach can be converted into mixed-integer linear programming (MILP) problems, which enables us to solve them with existing MILP solvers. We then show that the recursive feasibility of both the higher-level and the lower-level optimization problems can be guaranteed. In this way, we can achieve real-time train scheduling for the metro system. Numerical experiments, based on real-life data from the Beijing metro network, illustrate the effectiveness of the extended passenger absorption model and the proposed bi-level MPC approach.

Real-time train scheduling is essential for passenger satisfaction in urban rail transit networks. This paper focuses on real-time train scheduling for urban rail transit networks considering uncertain time-dependent passenger origin-destination demands. First, a macroscopic passenger flow model we proposed before is extended to include rolling stock availability. Then, a distributed-knowledgeable-reduced-horizon (DKRH) algorithm is developed to deal with the computational burden and the communication restrictions of the train scheduling problem in urban rail transit networks. For the DKRH algorithm, a cost-to-go function is designed to reduce the prediction horizon of the original model predictive control approach while taking into account the control performance. By applying a scenario reduction approach, a scenario-based distributed-knowledgeable-reduced-horizon (S-DKRH) algorithm is proposed to handle the uncertain passenger flows with an acceptable increase in computation time. Numerical experiments are conducted to evaluate the effectiveness of the developed DKRH and S-DKRH algorithms based on real-life data from the Beijing urban rail transit network. The simulation results indicate that DKRH can be used to achieve real-time train scheduling for the urban rail transit network, while S-DKRH can handle the uncertainty in the passenger flows with an acceptable sacrifice in computation time.

Effective timetable scheduling strategies are essential for passenger satisfaction in urban rail transit networks. Most existing passenger-centric timetable scheduling approaches generate a timetable according to deterministic passenger origin-destination (OD) demands. As passenger OD demands in urban rail transit networks generally show a high level of uncertainty, an effective timetable scheduling approach should take the uncertain passenger flows into account to generate a reliable timetable. In this paper, a scenario-based model predictive control (SMPC) approach is presented to handle uncertain passenger flows based on a passenger absorption model, where uncertainties are captured by several representative scenarios according to historical data. In each SMPC step, the optimization problem for generating the timetable can be reformulated as a mixed-integer linear programming (MILP) problem, which can be efficiently solved using current MILP solvers. A probabilistic performance level can be then determined based on the performance of SMPC under the representative scenarios. Numerical experiments based on the Beijing subway network are conducted to evaluate the efficacy of the proposed approach.

Real-time timetable scheduling is an effective way to improve passenger satisfaction and to reduce operational costs in urban rail transit networks. In this paper, a novel passenger-oriented network model is developed for real-time timetable scheduling that can model time-dependent passenger origin-destination demands with consideration of a balanced trade-off between model accuracy and computation speed. Then, a model predictive control (MPC) approach is proposed for the timetable scheduling problem based on the developed model. The resulting MPC optimization problem is a nonlinear non-convex problem. In this context, the online computational complexity becomes the main issue for the real-time feasibility of MPC. To reduce the online computational complexity, the MPC optimization problem is therefore reformulated into a mixed-integer linear programming (MILP) problem. The resulting MILP problem is exactly equivalent to the original MPC optimization problem and can be solved very efficiently by existing MILP solvers, so that we can obtain the solution very fast and realize real-time timetable scheduling. Numerical experiments based on a part of Beijing subway network show the effectiveness and efficiency of the developed model and the MILP-based MPC method.

This paper proposes a method to encourage safety in Model Predictive Control (MPC)-based Reinforcement Learning (RL) via Gaussian Process (GP) regression. The framework consists of 1) a parametric MPC scheme that is employed as model-based controller with approximate knowledge on the real system's dynamics, 2) an episodic RL algorithm tasked with adjusting the MPC parametrization in order to increase its performance, and 3) GP regressors used to estimate, directly from data, constraints on the MPC parameters capable of predicting, up to some probability, whether the parametrization is likely to yield a safe or unsafe policy. These constraints are then enforced onto the RL updates in an effort to enhance the learning method with a probabilistic safety mechanism. Compared to other recent publications combining safe RL with MPC, our method does not require further assumptions on, e.g., the prediction model in order to retain computational tractability. We illustrate the results of our method in a numerical example on the control of a quadrotor drone in a safety-critical environment.

Timetables determine the service quality for passengers and the energy consumption of trains in metro systems. In metro networks, a timetable can be made by designing train departure frequencies for different periods of the day, which is typically formulated as a mixed-integer linear programming (MILP) problem. In this paper, we first apply Benders decomposition to optimize the departure frequencies considering time-varying passenger origin-destination demands in metro networks. An ϵ -optimal Benders decomposition approach is subsequently used to reduce the solution time further. The performance of both methods is illustrated in a simulation-based case study using a grid metro network. The results show that both the classical Benders decomposition approach and the ϵ -optimal Benders decomposition approach can significantly reduce the computation time for the optimization of train departure frequencies in metro networks. In addition, the ϵ -optimal Benders decomposition approach can further reduce the solution time compared to the classical Benders decomposition approach when the problem scale increases while maintaining an acceptable level of performance.

Damping injection is a well-studied tool in nonlinear control theory to stabilize and shape the transient of mechanical systems. Interestingly, the injection of coupled damping yielding gyroscopic forces has received far less attention. This letter aims to fill this gap for gyroscopic forces that couple actuated and unactuated coordinates. First, we establish sufficient conditions for the stability of the closed loop. Then, we provide analytic results proving that injecting coupled damping may improve the closed-loop performance. We illustrate the results via the stabilization of three mechanical systems.