Trajectory prediction is a key element of autonomous vehicle systems, enabling them to anticipate and react to the movements of other road users. Robustness testing through adversarial methods is essential for evaluating the reliability of these prediction models. However, current approaches tend to focus solely on manipulating model inputs, which can generate unrealistic scenarios and overlook critical vulnerabilities. This limitation may result in incomplete assessments of model performance in real-world conditions. The specific effects of more comprehensive adversarial attacks on trajectory prediction models have not been thoroughly investigated. In this work, we demonstrate that by perturbing both model inputs and anticipated future states, we can uncover previously undetected weaknesses and provide a more realistic evaluation of model robustness. Our novel approach incorporates dynamical constraints and preserves tactical behaviors, enabling more effective and realistic adversarial attacks. We introduce new performance measures to assess the realism and impact of these adversarial trajectories. Testing our method on a state-of-the-art prediction model reveals significant increases in prediction errors and collision rates under adversarial conditions. Qualitative analysis further shows that our attacks can expose critical weaknesses, such as the model’s inability to detect potential collisions in what appear to be safe predictions. These results underscore the need for more comprehensive adversarial testing to better evaluate and improve the reliability of trajectory prediction models for autonomous vehicles. To support further research in this area, we provide an open-source framework for studying adversarial robustness in trajectory prediction. This work advances adversarial testing techniques, contributing to the safety and reliability of autonomous driving systems.

For trajectory prediction within autonomous vehicle planning and control, conditional variational autoencoders (CVAEs) have shown promise in accurate and diverse modeling of agent behaviors. Besides accuracy, explainability is also crucial for the safety and acceptance of learning-based autonomous systems, especially in autonomous driving. However, the latent distributions learned by CVAE models are often implicit and thus possess low explainability. To address this, we propose a semi-supervised generative modeling framework, \textbf{\textit{PrefCVAE}}, which utilizes partially and weakly labelled preference pairs to imbue the CVAE's latent representation with semantic meaning. This approach enables the system to estimate measurable attributes of the agents, and to generate manipulable predictions under the CVAE framework. Results show that incorporating our preference loss allows a CVAE-based model to make conditional predictions using the semantic factor of prediction average velocity. Our augmented framework also does not significantly degrade the baseline accuracy of prediction. Additionally, we show that the latent values learned with PrefCVAE better represent the semantic information contained in the data. Finally, we discuss the potential of this loss design to extend to other machine learning applications beyond trajectory prediction, as well as essential tricks for adaptation of human labeling. We hope that our empirical study offers the broader representation learning community a fresh perspective on inductive bias for disentangled and explainable latent representations in deep generative models. Specifically, we demonstrate that preference pair supervision, a simple and cost-effective approach, can effectively aid in learning semantic meanings for sampling-based generative models like the CVAE.

Autonomy in traffic (e.g., autonomous vehicles) could potentially benefit mobility, safety, accessibility and sustainability. However, the realisation of these advancements is highly dependent on how effective these autonomous vehicles interact with vulnerable road users such as pedestrians. Before we can understand how pedestrians will interact with autonomous vehicles, it is essential to understand how pedestrians interact among themselves in interactive traffic scenarios. Previous studies have focused on describing these scenarios with probabilistic trajectory prediction methods such as TrajFlow. However, these approaches often fall short in capturing the nuances of mutual interactions. Simple interaction models have been proposed that can describe these interactions, but neglect the influence of another person's intentions. To address this issue, in existing work the Communication-Enabled-Interaction (CEI) framework was proposed that describes interactions by modelling communication and a belief of another person's intentions. The idea of using beliefs in interaction modelling is based on the concept that people have a general but uncertain idea about the plans of other people. These beliefs are one of the fundamental aspects of the CEI framework and must therefore contain valuable information about possible decisions. That is why this study investigates the use of the probabilistic trajectory prediction method TrajFlow for the belief construction of the CEI framework. TrajFlow is trained on the belief-based Forking Paths dataset, integrated into the CEI framework, and tested in four simulated pedestrian interaction scenarios. The analysis shows that the framework is able to simulate plausible interaction behaviour, dealing with conflicting goals and trajectories in multiple simulations. By doing so, this study takes a positive step towards modelling pedestrian interactions and contributes to the broader goal of realising the benefits linked to autonomy in traffic.

This thesis explores enhancing track generalization in motorsport driver models through image-based feature sets, drawing inspiration from autonomous driving applications in urban settings. Traditional motorsport models often rely on numeric features, which excel on known tracks but face limita- tions when adapting to new, unseen environments. To address this, I introduce a CNN-based model that integrates bird’s- eye-view images with vehicle states and path-planning data, allowing a more holistic perception of track layouts and sur- roundings. Through open-loop evaluations on unseen tracks, the proposed model demonstrates superior generalization, achieving significantly lower RMSE compared to boundary point-based models, with improvements observed across steering, braking, and acceleration actions. Additionally, I apply novelty detection using Mahalanobis Distance to isolate Out-of-Distribution(OoD) scenarios, providing a precise measure of the generalization gap. This work establishes a baseline for image feature design in motorsport driver modeling, emphasizing the role of spatial and contextual information in achieving adaptable and high- performance autonomous racing agents.

Understanding traffic participants’ behaviour is crucial for predicting their future trajectories, enabling autonomous vehicles to better assess the environment and consequently anticipate possible dangerous situations at an early stage. While the integration of cognitive processes and machine learning models has demonstrated promise in various domains, its application in trajectory forecasting of multiple traffic agents in large-scale autonomous driving datasets remains lacking. This work investigates the state-of-the-art trajectory forecasting model Trajectron++ which we enhance by incorporating a smoothing term in its attention module. This attention mechanism mimics human attention inspired by cognitive science research indicating limits to attention switching. We evaluate the performance of the resulting Smooth- Trajectron++ model and compare it to the original model on various benchmarks. Our results show improved performance on the large-scale nuScenes dataset, revealing the potential of incorporating insights from human cognition into trajectory prediction models.

When a person makes a decision, it is automatically accompanied by a subjective probability judgement of the decision being correct, in other words, a (local) confidence judgement. Confidence judgements have, among other things, an
effect on justifications of future decisions and behaviour. A better understanding of the metacognitive processes responsible for these confidence judgements could improve behaviour models. To date, confidence judgements are mostly studied in a fundamental manner. Little to no research has been done into confidence in more dynamic tasks. Such applied research could render insights on whether fundamental principles also hold for real-life tasks. It could also have practical relevance for several applications. Driving is amongst the areas for which an improved understanding of the decision making and accompanied confidence judgements can be useful, for instance in order to improve driving assistance systems. However, current studies on driving behaviour are merely focused on decision making and do not take confidence into account.
In this study, we made a first attempt of connecting these two fields of research by investigating the confidence of drivers in left-turn gap acceptance decisions in a driver simulator experiment (N=17). The study showed that confidence can be related to the gap size with respect to the oncoming vehicle, described by the time-to-arrival and the distance gap. Confidence increases with the gap size for gap accepting decisions and decreases with the gap size for gap rejecting decisions. In addition, we concluded that confidence can be related to the driving behaviour, and that confidence is negatively related to the decision response time. Moreover, we found that confidence judgements can best be captured with the use of an extended dynamic drift diffusion decision model of which the drift rate of the evidence accumulator as well as the decision boundaries are functions of the time-to-arrival and distance gap. Furthermore, we demonstrated that allowing for post-decision evidence accumulation in the model increases its ability to describe confidence judgements in gap rejecting decisions. Overall, the study confirmed that principles known from fundamental confidence research can be used to describe confidence judgements in a dynamic and applied task.

The rapid advancement in autonomous driving technology underscores the importance of studying the fragility of perception systems in autonomous vehicles, particularly due to their profound impact on public transportation safety. These systems are of paramount importance due to their direct impact on the lives of passengers and pedestrians. Additionally, their reliability can be easily compromised given the complexity and unpredictability of driving environments. However, current research and existing regulations often fail to adequately address the adversarial robustness of autonomous vehicle perception systems. This thesis delves into the adversarial robustness of camera-based perception systems of autonomous vehicles. Our research concentrates on developing and implementing evasion attacks that use black-box gradient estimation, as well as physical attacks in traffic sign detection and classification systems. Our findings indicate that even minor perturbations can impact the accuracy of these systems, leading to detection and classification errors. This finding highlights a critical vulnerability in the perception system's robustness against adversarial attacks. Moreover, the study extends to assess the transferability of adversarial examples across diverse perception models. Our results also expose significant gaps in the current regulatory frameworks of autonomous vehicles, necessitating the establishment of more rigorous and comprehensive safety standards.

Understanding human behavior in overtaking scenarios is crucial for enhancing road safety in mixed traffic with automated vehicles (AVs). Modeling plays a pivotal role in advancing our comprehension of human overtaking behavior in dynamically evolving scenarios. Currently, our understanding of overtaking behavior primarily revolves around straightforward interactions with human-driven vehicles (HDVs). To address this gap, we conducted a ``reverse" Wizard-of-Oz driving simulator experiment with 30 participants interacting with both oncoming AVs and HDVs, featuring time-varying dynamics. We hypothesized that the type of oncoming vehicle (AV or HDV) does not significantly influence gap acceptance during overtaking, while we anticipated an increase in gap acceptance when the oncoming vehicle briefly decelerates during interactions with the human ego-vehicle driver. Our findings reveal that participants did not significantly alter their overtaking behavior when interacting with oncoming AVs compared to HDVs. Surprisingly, brief decelerations in the oncoming vehicle's velocity did not significantly affect the decision-making processes of overtaking. Moreover, our results reinforced previous insights into the significance of the initial distance and time-to-arrival to the oncoming vehicle, and the ego-vehicle velocity on participants' overtaking behavior. We highlight the potential of simple drift-diffusion models (DDMs), a subset of cognitive models, in understanding human overtaking behavior in dynamically evolving scenarios involving oncoming AVs. Our proposed model accurately captures qualitative patterns in gap acceptance during these intricate overtaking scenarios, further advancing the ongoing development of safer interactions between human drivers and AVs during overtaking maneuvers.

In order to design safe and effective interactions between autonomous vehicles (AVs) and human road users, it is essential to understand the mechanisms underlying human-human merging behavior. Driving simulator experiments can be used to study these mechanisms, but previous research has primarily focused on the behavior of individual drivers rather than the dynamics of interactions. In addition, current experimental scenarios and data analysis tools do not adequately capture interactive humanhuman merging behavior. To address these issues, I propose an experimental framework featuring a simplified highway-merging scenario that can facilitate human factors research on merging interactions. In a case study with fourteen participants, I used the framework in a coupled virtual reality driving simulator to show a relation between participants’ interactive behavior and fixation behavior. This work shows how to better understand human-human merging interactions, which is essential for developing AVs that can safely and successfully interact with other road users.

Background: Merging on a highway is a complex driving task that requires a lot of interaction with other road users. During these tasks, a driver is required to evaluate gaps in space and time between the themselves and other road users and obstacles in order to arrive at the right moment to merge onto the highway. To improve safety and increase road efficiency, it is necessary to understand the decision process during merging decisions. A way of achieving this, is to understand what visual information humans use during this decision process. This study investigated the relation between gaze location and gap acceptance decisions during highway merges.
Methods: An experiment was performed in which 26 participants monitored an automated vehicle (AV) that was driving on a highway on-ramp. The participants were given the task to train the AV in whether or not to merge in front of an upcoming vehicle that was already driving on the highway. An eye tracker was used to measure gaze data, which was used to find the relation between gaze behaviour and decision outcomes and response times. A mixed-effects logistic model was used for a statistical analysis with decision outcomes as a dependent variable and different gap sizes as predictor variables. A mixed-effects linear model was used to find the relation between response times and dwell times and the different gap sizes and decision outcomes as predictor variables. For both the decision outcome and response time model, dwell time was later included to find the effect on the predictive validity.
Results: The results show that a larger time and distance gap to the upcoming vehicle relate to a higher merging probability. For larger time gaps to the on-ramp, the probability of merging was found to be smaller. It was also found that time gaps to the end of the on-ramp significantly relate to response times, with an increase of 55ms per 1s. Larger time gaps to the upcoming vehicle significantly relates to larger response times, with an increase of 64ms per 1s. No significant relation was found between response time and distance gaps to the upcoming vehicle. The response time was found to be 0.60s longer for rejected gap decisions. The time gap to the end of the on-ramp significantly relates to dwell time, with an increase of 0.56% per 1s. The distance gap to the upcoming vehicle significantly relates to dwell time, with an increase of 0.60% per 10m. The time gap to the upcoming vehicle significantly relates to dwell time, with an increase of 0.52% per 1s. The presented results show as well that a significant relation exists between gaze behaviour and decision outcomes and response times. When analysing decision outcomes and response times, the interaction between dwell time and gap sizes should be taken into account. This improved the predictive validity of the used regression models.
Conclusion: Several pieces of evidence suggest that gaze behaviour assist in understanding the human decision making process during merging. This study can serve as a basis for cognitive models that can investigate how the relation between gaze behaviour and gap sizes, decision outcomes and response times can help to understand and potentially predict gap acceptance decisions.

A common method to investigate multisensory integration is using multisensory illusions. The rubber hand illusion is one of the best-known multisensory illusion used in clinical applications. By stroking a visible rubber hand and the participant’s occluded hand, the illusion arises that the rubber hand belongs to the participant. Possible applications based on this illusion are for neurorehabilitation or developing robotic devices. These applications are still in a very initial state, and this thesis aims to take these possible applications a step further. The thesis provides the results of a rubber hand illusion experiment over time and a dynamic model related to the embodiment of the rubber hand. The dynamical model can visualize the proprioceptive drift that arises from the illusion and predict the time it takes to experience it. The results of the experiment and other empirical findings from the literature form the basis of the model. The experiment measured these factors and the feeling of ownership and agency over time. The results from this experiment have much variance but are in line with the literature. The dynamical model fails to visualize the body ownership and agency results. Adding extensions and improvements could make this dynamic model more complete and applicable in many research fields.

Video Object Detectors (VID) are used in various applications such as surveillance, inspection, etc. Often in these applications there exists a spatial area of interest and a static background. The static backgrounds remain constant throughout the video sequence in the training data establishing an undesirable correlation with the moving object during training. To hide static backgrounds in the video, masking is an option. We create multiple synthetic datasets and reveal that
(i) VIDs detect moving objects better if the static background in the train and test set are similar or from the same distribution.
(ii) VIDs drop in performance if the static background in the train and test are different.
(iii) Adding more static backgrounds during training does not make VID robust to static background changes at test time.
(iv) Masking or removing static backgrounds cannot prevent VIDs from learning correlations with static backgrounds.
The experiments shed light on the usage of static backgrounds for detecting dynamic objects.

Overtaking on two-lane roads can cause dangerous situations, due to drivers’ errors during the gap acceptance decision. Understanding gap acceptance decisions can help mitigate these situations. Response time (i.e. the time it takes the driver to evaluate the gap and make a decision) has been shown to provide valuable insights into the cognitive processes during gap acceptance decisions in pedestrian crossing and left turn decisions. However, for overtaking, previous studies have not investigated the response time.
I investigated how factors that play a role in gap acceptance decisions during overtaking (namely, size of the gap and the drivers’ velocity) influence the response time. I also studied the change of the drivers' velocity during the decision process. I proposed a novel method to measure the response time in drivers' overtaking decisions, and conducted an experiment to demonstrate the usefulness of the method. 25 participants were presented with multiple overtaking situations in a driving simulator experiment, with varying distance gaps. I analyzed how the probability of gap acceptance varied with the distance gap and participants’ velocity at the start of the overtaking situation using a generalized logistic mixed model. I also analyzed how the response time varied with the distance gap, participants’ velocity and the decision outcome using a linear mixed model. Lastly, I analyzed if the velocity changed between the start and the end of the decision process and whether there was a relation between the decision outcome and the distance gap and the velocity change, using a linear mixed model.
The probability of accepting a gap increased significantly with the distance gap and the velocity of the participant. The response times for rejected gaps were on average 0.7s longer than accepted gaps. The response time increased with the distance gap (42ms per 10m), but decreased with the velocity (-92ms per 1m/s). The velocity changed differently between the decision processes leading to either decisions, with an average difference of 4 m/s.
Using the proposed method, I found that the factors which influence the outcome of the decision, also influence the response time. The dependence of response time on the distance gap and participants’ velocity could be explained by the speed-accuracy tradeoff or the difficulty of the decision. Furthermore, it was shown that the drivers already adapt their velocity during the decision process, instead of after they have made their decision. So, I can conclude that using the proposed method for measuring response time can give insight in the way drivers make gap acceptance decisions during overtaking. My results provide basis for cognitive process models that can help further understand the results and are capable of predicting decision outcomes and response times.

Traffic jams occurring on highways cause increased travel time as well as increased fuel consumption and crashes. Traffic jams without a clear cause, such as an on-ramp or an accident, are called phantom traffic jams and are said to make up 50% of all traffic jams. They are the result of an unstable traffic flow caused by human driving behaviour. Recent studies have shown how automating the longitudinal vehicle motion of only 5% of all cars in the flow can dissipate phantom traffic jams. However, automation introduces new problems, mainly regarding safety when human drivers need to take over the control. This research tries to answer whether phantom traffic jams can be dissolved or prevented using haptic shared control. This means of control keeps the human in the loop and would therefore eliminate the takeover problems while still benefiting from the advantages of automation. 24 participants took part in a driving experiment in a fixed base simulator. In these experiments, we tested haptic shared control against manual control and full automation for longitudinal motion. Results show that traffic jam dissipation performance for haptic shared control lies between manual control and automation. The number of unsafe situations is reduced compared to the automated condition. We conclude that haptic shared control is able to reduce the increased fuel consumption and crashes caused by phantom traffic jams.

The field of deep reinforcement learning has seen major successes recently, achieving superhuman performance in discrete games such as Go and the Atari domain, as well as astounding results in continuous robot locomotion tasks. However, the correct specification of human intentions in a reward function is highly challenging, which is why state-of-the-art methods lack interpretability and may lead to unforeseen societal impacts when deployed in the real world. To tackle this, we propose multi-objective reinforced active learning (MORAL), a novel framework based on inverse reinforcement learning for combining a diverse set of human norms into a single Pareto optimal policy. We show that through the combination of active preference learning and multi-objective decision-making, one can interactively train an agent to trade off a variety of learned norms as well as primary reward functions, thus mitigating negative side effects. Furthermore, we introduce two toy environments called Burning Warehouse and Delivery, which allow for studying the scalability of our approach in both size of the state space and reward complexity. We find that through mixing expert demonstrations and preferences, we can achieve superior efficiency compared to employing a single type of expert feedback and, finally, suggest that unlike previous literature, MORAL is able to learn a deep reward model consisting of multiple expert utility functions.

When humans make inferences that go beyond limited, noisy, or ambiguous input data, background knowledge is necessary to make generalizations. Such inferences are important for designing intelligent artificial agents. Bayesian inference, a statistical method commonly used as a cognitive model of the brain, formalizes the integration of background knowledge, i.e. "priors", and sensory evidence. I research the effect of priors on sense of agency (SoA) in an artificial agent. In humans, SoA is the subjective experience of control over actions and their consequence. In robotics, developing an agent with SoA is a popular challenge and the first step towards the artificial self.

This thesis designs an artificial agent where the same prior knowledge improves SoA in unambiguous environments and induces incorrect, illusory SoA in noisy, ambiguous environments. First, I provide a comprehensive overview on the role of priors in Bayesian inference and the computational principles of a SoA. Second, I define the ``point mass moving rubber hand illusion (mRHI)", a simulation for an artificial agent that simplifies the human mRHI experiment. Third, I use two parameter estimators, ordinary least-squares (OLS) and expectation-maximization (EM), to calculate SoA in the point mass mRHI.

I found that SoA requires a prior belief about the causal relationship between outcome and action. In the point mass mRHI, the agent has to identify which of the three point masses it can control with its force input. The agent has the prior knowledge that there is a causal relationship between its actions and the states of one point mass. Though with OLS the agent has correct and incorrect SoA, some of the results not comparable to the results from human mRHI experiment. A likely cause is that this simplest reproduction of the mRHI models SoA as a binary variable and does not use Bayesian inference. The (partially) Bayesian parameter estimator EM calculates SoA as a (continuous) posterior probability and finds results similar to the human mRHI experiment. Comparing an OLS algorithm without prior to the EM algorithm, I found that in unambiguous environments the Bayesian prior improves the agent's general SoA and SoA over time but does not improve the SoA for the point mass that the agent does control. In the ambiguous environments, the prior generally motivates the agent towards a SoA over the incorrect mass. However, in a noisy but slightly less ambiguous environment, the prior improves the agent's SoA. In short, the prior improves a SoA when the agent needs to make inferences that go beyond the data but can also induce an incorrect SoA in a noisy and ambiguous environment.