Rapid and continuous advances in communications and computer technology are spurring
a host of new concepts in road traffic control. From simple traffic control measures like
lane segregation for wheeled and pedestrian traffic in the 14th century to the use of artificial
intelligence to control traffic, we have come a long way. The advancement of technology in
the automotive sector along with the industrial revolution led to the number of vehicle owners
seeing a drastic increase in the 20th century. There was a growing need for infrastructure to
handle this new volume of vehicles which led to the construction of several road networks in
both urban and freeway settings. Adding infrastructure was a valid method to handle traffic
problems until we realized that each new addition required a significant amount of land that
was slowly reducing. This brought along the need for newer and more efficient traffic control
techniques due to increased vehicles on the road, the different kinds of vehicles itself and the
need to reduce construction of roads as a means of reducing traffic.
In this thesis report, we discuss the different control methods using reinforcement learning
to tackle the freeway traffic control problem. The thesis covers the fundamentals of freeway
traffic control, reinforcement learning and the agents used for control. It focuses on the
creation of the freeway network, environment setup for reinforcement learning application
and the choice of agents mainly SAC in order to implement continuous actions to improve
combined ramp metering and variable speed limit control in more complex scenarios

To ensure safe operation of autonomous vehicles (AVs), trajectory planners should account for occlusions. These are areas invisible to the AV that might contain vehicles. Set-based methods can guarantee safety by calculating the reachable set, which is the set of possible states, for each potentially hidden vehicle. A recently published method proved in simulation experiments to reduce the cautiousness by reasoning about these occluded areas over time, assuming perfect input data. We present a novel algorithm that uses this reasoning and is applicable on a real AV with its accompanying uncertainties and imperfect sensor data. The uncertainties include sensor errors and noise, computation and communication delays and control errors in the trajectory following. This is achieved by modelling the error distributions and accounting for them in the calculations, where the confidence interval for each error is exposed as a setting. Experiments indicate that our algorithm can reduce the traversal time through an intersection by 2.2 seconds with reasoning. An ablation study of the different error measures shows that the errors in the construction of the field of view (FOV) limit the performance the most. Reducing the errors in the FOV construction is therefore the most important recommendation, besides making the method interaction-aware.

The renewable generation capacity, particularly solar and wind power installations, has increased steadily in the Netherlands over the course of recent years. Due to the local, small-scale nature of these power plants (compared to conventional power plants), a large share of this generation capacity is installed into the medium- and low-voltage distribution grids. This trend acts as the source of several challenges for distribution system operators (DSOs) such as Stedin Netbeheer B.V. (a DSO in the Netherlands, supporting this project). One of the main problems is the fact that voltage limit violations, particularly upper voltage limit violations, become more frequent.
Conventional voltage control schemes assume a unidirectional power flow (i.e., consumption only) in distribution grids and are unable to keep the limits in case a large share of generation capacity is installed. This is further complicated by the fact that underground cable networks, typical in Western European electricity distribution grids, have lines with large R/X ratios, which reduces the effectiveness of reactive power injection-based voltage control methods.
This MSc project intends to solve the issue of voltage limit violations with a model predictive control (MPC) policy. The considered control actions coordinated by the model predictive controller are the switching of the on-load tap changer (OLTC) mechanism mounted to the primary substation’s transformer, setpoint adjustments of the low-level OLTC control relay, and the active power curtailment of larger photovoltaic plants. A linear, sensitivity-matrix-based model is used for the grid’s state prediction; and the sensitivity values are re-calculated at each sampling time step of the MPC. To avoid the curtailment of photovoltaic (PV) plants when not justified, a conditional curtailment logic is incorporated into the MPC policy: PV plants are only allowed to be curtailed if their local voltage magnitude is above a tuneable threshold. This logic is described and incorporated into the model predictive controller’s optimization problem with the help of binary variables and mixed-integer linear (MIL) constraints. The benefit of incorporating knowledge about future disturbances (load, generation, and external grid voltage profiles) is also tested, in order to assess the potential benefit DSOs could get from forecasting these quantities.
A case study was conducted in which the considered controllers were tested on a section of a Stedin grid that carries the characteristics of a typical Western European medium voltage distribution grid: large R/X ratios and a large installed PV generation capacity both in the form of household generation and larger PV plants. All controllers were simulated in 4 different test cases: a typical summer day, a summer day with the external grid’s overvoltage, a summer day with 2 out of 3 large PV plants not operating, and a typical winter day. All test cases have the control goals of mitigating limit violations and ensuring that the nodal voltage magnitudes are as close as possible to the nominal 1 per unit throughout the day. The winter day test case has the additional goal of avoiding excessive curtailment as PV energy is worth considerably more during these days. All profile data is based on real Stedin measurements. The designed model predictive control policies are compared to two simple control schemes: current compounding, i.e. when the primary substation’s automatic voltage control relay’s setpoint is adjusted based on the active power delivery through the substation’s transformer, and another scheme when current compounding is combined with local active power curtailment controllers for large PV plants. The most important metrics used for comparison are voltage root mean square error (RMSE), the total voltage limit violation area, the percentage of curtailed PV energy, and the number of tap changes over the considered day. The simulations were carried out using Python and DIgSILENT PowerFactory.
The simulation results show that the MPC policy can perform better than the simple control schemes but only when exact knowledge of future profiles is available. In this case, the MPC results in lower voltage RMSE, smaller violation areas, and lower curtailment percentage values, at the expense of using more tap changes in all 4 test cases. In 3 of the 4 test cases, MPC was completely able to eliminate voltage limit violations, showing clearly the advantages of good quality forecasts on future disturbances. Since this exact knowledge about the near future fluctuations is quite ideal, more realistic MPC policies were also tested with no future knowledge and tightened voltage constraints. These simulations brought mixed results when compared with the simple schemes, performing better in terms of voltage RMSE, but worse in terms of the total voltage limit violation area.