The ongoing increase in urbanization and traffic congestion creates an urgent need to operate our transportation systems with maximum efficiency. Traffic signal control optimization is considered one of the main ways to solve traffic problems in urban networks. In publications in the field of intelligent transportation systems, a vast amount of different optimal traffic light control methods is described. With new optimization methods being developed, it is important to know their performance compared to similar methods. Such a comparison is only possible if the same performance evaluation methodology is applied to all these methods. Most of the studies in the field of intelligent transportation system consider a self-defined evaluation methodology. A general evaluation methodology is developed to objectively evaluate the performance of these optimization methods. The developed general evaluation methodology is used to evaluate the performance of a dynamic programming and Q-learning method.

The Decentralized Partially Observable Markov Decision Process is a commonly used framework to formally model scenarios in which multiple agents must collaborate using local information. A key difficulty in a Dec-POMDP is that in order to coordinate successfully, an agent must decide on actions not only using its own information, but also by reasoning about the information available to the other agents. Nevertheless, existing value-based Reinforcement Learning techniques for Dec-POMDPs typically take the individual perspective, under which each agent optimizes its own actions using solely its local information, thus essentially neglecting the presence of others. As a result, the concatenation of individual policies learned in this way has a tendency to result in a sub-optimal joint policy. In this work, we propose to additionally condition such Independent Q-Learners on the plan-time sufficient statistic for Dec-POMDPs, which contains a distribution over the joint action-observation history. Using this, the agents can accurately reason about the resulting actions the other agents will take, and adjust their own behavior accordingly. Our main contributions are threefold. (1) We thoroughly investigate the effects of conditioning Independent Q-Learners on the sufficient statistic for Dec-POMDPs. (2) We identify novel exploration strategies that the agents can follow by conditioning on the sufficient statistic, as well as their implications on the decision rules, the sufficient statistic and the learning process. (3) We substantiate and demonstrate that by conceptually sequencing the decision-making, and additionally conditioning the agents on the current decision rules of the earlier agents, such learners are able to consistently escape sub-optimal equilibria and learn the optimal policy in our test environment, Dec-Tiger.

Even though the abaility to recommend items in the long tail is one of the main strengths of recommendation systems, modern models still show decreased performance when recommending these niche items. Various bipartite and tripartite graph-based models have been proposed that are specifically tailored to solving this long tail issue. This study aims to investigate the effect of the design of the additional layer introduced by tripartite graph-based recommender systems on their performance. All options available in the MovieLens 1M dataset are evaluated on recall and diversity. Experimental results suggest that tripartite graphs based on latent information describing the users perform better than ones utilising item-based latent information, but both these options hardly outperform the baseline bipartite model. Regardless of the graph used, normalising the transition matrix is found to significantly increase performance. It is hypothesised that larger user-focused additional layers show increased diversity over smaller options when normalised. Issues regarding the reproducibility of previous research are identified and addressed, and the development of unified evaluation metrics is advocated to prevent such problems in the future.

Recommender Systems play a significant part in filtering and efficiently prioritizing relevant information to alleviate the information overload problem and maximize user engagement. Traditional recommender systems employ a static approach towards learning the user's preferences, relying on logged previous interactions with the system, disregarding the sequential nature of the recommendation task and consequently, the user preference shifts occurring across interactions. In this study, we formulate the recommendation task as a slate Markov Decision Process (slate-MDP) and leverage deep reinforcement learning (DRL) to learn recommendation policies through sequential interactions and maximize user engagement over extended horizons in non-stationary environments. We construct the simulated environment with various degrees of preferential dynamics and benchmark two DRL-based algorithms: FullSlateQ, a non-decomposed full slate Q-learning based on a DQN agent, and SlateQ, which implements DQN using slate decomposition. Our findings suggest that SlateQ outperforms by 10.57% FullSlateQ in non-stationary environments and that with a moderate discount factor, the algorithms behave myopically and fail to make an appropriate tradeoff to maximize long-term user engagement.

One of the most important bottlenecks that contributes to the congestion of traffic is nonoptimal traffic signal control. Techniques that have been investigated to optimise traffic signal control have been focused on improving the traffic flow through individual intersections. However, if intersections are optimised based on only reducing local congestion, this could result in introducing congestion in other places. Therefore, this paper introduces a normal form game that can be used to analyse the impact on local and global performance when optimising traffic signal control. Results obtained from a realistic traffic simulation suggest that it is sometimes possible for each player to choose a traffic signal control strategy that optimises its own welfare but also maximises the social welfare. The results also indicate that sub-optimal traffic signal control strategy profiles can become optimal when exposed to certain traffic intensities.

Recommender systems are an essential part of online businesses in today's day and age. They provide users with meaningful recommendations for items and products. A frequently occurring problem in recommender systems is known as the long-tail problem. It refers to a situation in which a majority of the items in the data set have limited ratings due to which many recommender systems, especially collaborative filtering based methods, are not able to recommend these items, also known as long-tail items. Although popular items are easier to recommend, it has been noticed that long-tail items often generate a significant fraction of the revenue and therefore should also be recommended to users. This paper proposes a modified version of a collaborative filtering based recommender system aimed to reduce the effects of the long-tail recommendation problem (LTRP). The algorithm first splits the data set into the head H and the tail T and clusters the items from the tail. The average rating avg for each cluster is calculated and for all users and their unrated long-tail items, the rating for that item is set to avg with a probability of p. Now the standard collaborative filtering algorithm is run with the newly inserted ratings. The inserted ratings reduce the sparsity of the data set and therefore make it easier to recommend long-tail items. Empirical experiments on the 100K MovieLens data set indicate that the proposed algorithm recommends more long-tail items than the standard collaborative filtering algorithm, thus reducing the effects of the LTRP while maintaining the same or a slightly lower accuracy of the recommender system.

Optimization of traffic signal control has been widely investigated by means of model-based strategies. In 2012 a new model-based controller was published, named Schedule-driven Intersection Control (SCHIC). This controller uses a job-scheduling algorithm to minimize the cumulative delay for all observed vehicles. The algorithms of SCHIC are at the basis of the commercially deployed controller SURTRAC. This means that SCHIC is partly used in real-world traffic situations and its algorithms are well documented. Therefore, it was possible to reproduce this controller in this thesis. Previous literature often evaluated SCHIC in simplified scenarios with (1) unrealistically large look-ahead times for vehicle detection. Secondly, (2) in most scenarios all approaching vehicles were restricted to request a single turning-movement at the intersection thereby removing inter-phase uncertainty (as explained directly below). Inter-phase uncertainty occurs when the chosen entry-lane of a vehicle determines the green phase that it requires from the traffic light controller. So, inter-phase uncertainty occurs under the following two conditions. Condition 1 – an approaching vehicle must eventually choose between different entry-lanes, with each entry-lane being associated to a different turning-movement at the intersection. Condition 2 – each of these individual entry-lanes is serviced during a different and competing green phase. For example, inter-phase uncertainty occurs when two competing green phases service either the straight-ahead or the left-turning entry-lane. The aim of this thesis is to evaluate SCHIC with both (1) realistic look-ahead times and (2) inter-phase uncertainty, since these two uncertainties are often not found in previous literature, however, they are commonly found in reality. Thus, it is not directly clear how the reported results would translate to more realistic settings, which include these two common uncertainties. In this study SCHIC is re-implemented and evaluated on a typical Dutch urban intersection located in The Hague. The lane dimensions of this intersection determine the look-ahead times as used in this thesis. For this study, a new method is introduced to reduce inter-phase uncertainty which is inspired by lane change prediction modules for automated driving. This method, Lane Change Prediction (LCP), makes use of e.g. radar-equipment in order to observe: (1) the driving style of each approaching vehicle and (2) the upcoming traffic state on each reachable entry-lane. Based on these two observations a classification model, i.e. a Random Forest, predicts the desired entry-lane before the vehicle actually enters it. This thesis researches whether the potentially higher accuracy of LCP will improve SCHICs performance as compared to its simpler statistical baselines, fixed-label and split-vehicles, which could have a lower accuracy. All traffic scenarios are simulated with Simulation of Urban Mobility (SUMO). Two popular driver-models are evaluated which determine the dynamic behaviour of the vehicles in simulation, i.e. the Intelligent Driver Model (IDM) and Krauss driver-model. The results showed that despite its higher accuracy, LCP does not significantly improve SCHICs performance over its baseline methods fixed-label and split-vehicles. Furthermore, the maximum theoretical performance gain of using LCP is evaluated with perfect predictions (Oracle). When applying perfect predictions, the average speed of all vehicles increased by 0.65% and 0.51% for the Intelligent Driver Model (IDM) and Krauss driver-model, respectively. The average number of stops per vehicle was reduced by 1.53% and 1.12% for the IDM and Krauss driver-model, respectively. Overall, it is concluded that the performance potential of all methods, i.e. the {Oracle, split-vehicles, fixed-label and LCP}, is quite comparable on SCHICs traffic flow. These methods show some performance potential, however, their confidence intervals are relatively wide which draws them closer to each other. In our scenario, the intersection in The Hague, another method might increase SCHICs traffic flow more significantly.

Recommender systems (RS) are a cornerstone for most online businesses that cater to a large customer base such as e-commerce, social network platforms and many others. RS's enable these platforms to provide tailor-made experiences to each of their customers by strategically utilizing users/items rating data or any other available data. Collaborative filtering (CF) techniques are some of the most popular and successful RS models created. However, CF techniques often suffer from the cold start (CS) problem. In particular, they struggle with complete cold start (CCS) situations in which no user/item rating history is available and incomplete cold start (ICS) situations in which only a limited amount of user/item rating history is available. In this paper, we explore two models which utilize novel ideas to combat the CCS and ICS problems. The first model (DCF) focuses on the intelligent use of user demographic data to combat the CCS problem. The second model (PIPCF) focuses on the use of a novel domain-specific similarity measure called Proximity-Impact-Popularity (PIP) to combat the ICS problem. In addition to this, we also propose our own model (DPIP-CF) which combines these two ideas in conjunction with some of our own modifications to combat the CCS and ICS problems simultaneously. We utilize the MovieLens data set which is a commonly available and popular dataset that is often used to test RS's. Through a series of experiments, we demonstrate the strengths of DCF and PIPCF in dealing with the CCS and ICS problems respectively. Finally, we also show that our DPIP-CF model outperforms all other models discussed in this paper and is a viable solution to dealing with the CCS and ICS problems simultaneously.