Optimization of traffic signal control has been widely investigated by means of model-based strategies. In 2012 a new model-based controller was published, named Schedule-driven Intersection Control (SCHIC). This controller uses a job-scheduling algorithm to minimize the cumulative delay for all observed vehicles. The algorithms of SCHIC are at the basis of the commercially deployed controller SURTRAC. This means that SCHIC is partly used in real-world traffic situations and its algorithms are well documented. Therefore, it was possible to reproduce this controller in this thesis. Previous literature often evaluated SCHIC in simplified scenarios with (1) unrealistically large look-ahead times for vehicle detection. Secondly, (2) in most scenarios all approaching vehicles were restricted to request a single turning-movement at the intersection thereby removing inter-phase uncertainty (as explained directly below). Inter-phase uncertainty occurs when the chosen entry-lane of a vehicle determines the green phase that it requires from the traffic light controller. So, inter-phase uncertainty occurs under the following two conditions. Condition 1 – an approaching vehicle must eventually choose between different entry-lanes, with each entry-lane being associated to a different turning-movement at the intersection. Condition 2 – each of these individual entry-lanes is serviced during a different and competing green phase. For example, inter-phase uncertainty occurs when two competing green phases service either the straight-ahead or the left-turning entry-lane. The aim of this thesis is to evaluate SCHIC with both (1) realistic look-ahead times and (2) inter-phase uncertainty, since these two uncertainties are often not found in previous literature, however, they are commonly found in reality. Thus, it is not directly clear how the reported results would translate to more realistic settings, which include these two common uncertainties. In this study SCHIC is re-implemented and evaluated on a typical Dutch urban intersection located in The Hague. The lane dimensions of this intersection determine the look-ahead times as used in this thesis. For this study, a new method is introduced to reduce inter-phase uncertainty which is inspired by lane change prediction modules for automated driving. This method, Lane Change Prediction (LCP), makes use of e.g. radar-equipment in order to observe: (1) the driving style of each approaching vehicle and (2) the upcoming traffic state on each reachable entry-lane. Based on these two observations a classification model, i.e. a Random Forest, predicts the desired entry-lane before the vehicle actually enters it. This thesis researches whether the potentially higher accuracy of LCP will improve SCHICs performance as compared to its simpler statistical baselines, fixed-label and split-vehicles, which could have a lower accuracy. All traffic scenarios are simulated with Simulation of Urban Mobility (SUMO). Two popular driver-models are evaluated which determine the dynamic behaviour of the vehicles in simulation, i.e. the Intelligent Driver Model (IDM) and Krauss driver-model. The results showed that despite its higher accuracy, LCP does not significantly improve SCHICs performance over its baseline methods fixed-label and split-vehicles. Furthermore, the maximum theoretical performance gain of using LCP is evaluated with perfect predictions (Oracle). When applying perfect predictions, the average speed of all vehicles increased by 0.65% and 0.51% for the Intelligent Driver Model (IDM) and Krauss driver-model, respectively. The average number of stops per vehicle was reduced by 1.53% and 1.12% for the IDM and Krauss driver-model, respectively. Overall, it is concluded that the performance potential of all methods, i.e. the {Oracle, split-vehicles, fixed-label and LCP}, is quite comparable on SCHICs traffic flow. These methods show some performance potential, however, their confidence intervals are relatively wide which draws them closer to each other. In our scenario, the intersection in The Hague, another method might increase SCHICs traffic flow more significantly.

The Decentralized Partially Observable Markov Decision Process is a commonly used framework to formally model scenarios in which multiple agents must collaborate using local information. A key difficulty in a Dec-POMDP is that in order to coordinate successfully, an agent must decide on actions not only using its own information, but also by reasoning about the information available to the other agents. Nevertheless, existing value-based Reinforcement Learning techniques for Dec-POMDPs typically take the individual perspective, under which each agent optimizes its own actions using solely its local information, thus essentially neglecting the presence of others. As a result, the concatenation of individual policies learned in this way has a tendency to result in a sub-optimal joint policy. In this work, we propose to additionally condition such Independent Q-Learners on the plan-time sufficient statistic for Dec-POMDPs, which contains a distribution over the joint action-observation history. Using this, the agents can accurately reason about the resulting actions the other agents will take, and adjust their own behavior accordingly. Our main contributions are threefold. (1) We thoroughly investigate the effects of conditioning Independent Q-Learners on the sufficient statistic for Dec-POMDPs. (2) We identify novel exploration strategies that the agents can follow by conditioning on the sufficient statistic, as well as their implications on the decision rules, the sufficient statistic and the learning process. (3) We substantiate and demonstrate that by conceptually sequencing the decision-making, and additionally conditioning the agents on the current decision rules of the earlier agents, such learners are able to consistently escape sub-optimal equilibria and learn the optimal policy in our test environment, Dec-Tiger.