Numerous techniques have been developed in order to explain the reasoning process of black-box models. Among them is a class of models that are designed to be inherently interpretable: select-then-predict models (a.k.a. rationale-based models). These models are meant to explain their prediction by highlighting part of the input as evidence. The evidence, called the rationale, should consist of the most salient parts of the input text that contribute the most to the model's decision.
However, according to some recent studies, these models are not truly interpretable, because they do not provide faithful explanations (i.e., explanations that accurately reflect the true reasoning process of the model).
In this thesis we give a formal definition of the degree of unfaithfulness to quantify unfaithful behavior. Then, we introduce an experiment to test the faithfulness of select-then-predict models and prove that select-then-predict models can provide unfaithful rationales. Lastly, we introduce a loss function, which we call the unfaithfulness loss, which minimizes the degree of unfaithfulness of select-then-predict models and teaches them to produce more faithful and plausible rationales.

A large number of traffic accidents occur worldwide each year, of which a sizable portion involve pedestrians, making them a vulnerable group on the road. Many of these accidents occur due to visual distraction, meaning drivers and pedestrians fail to look where they should be looking. In addition to this tendency for distraction, the eyes are a means of exchanging information between road users, via behaviors such as eye contact. However, the role and importance of eye contact in traffic in connection with traffic safety and the decisions of road users is not yet entirely clear. Further, with the advent of automated vehicles, the role of eye contact in traffic may change or disappear altogether, due to the absence of drivers. One promising way to shed light on this matter is to use eye-tracking, a technology which can measure the eye movements of road users, and which might allow the engineering of solutions to mitigate the frequency and severity of accidents.

This dissertation aims to investigate the role of eye contact between drivers and pedestrians, as well as its influence on pedestrians’ road crossing intentions. Another aim of this dissertation is to assess the accuracy of eye-tracking devices and to objectively detect and operationalize driver-pedestrian eye contact using eye-tracking. Finally, this thesis aims to develop safety systems based on eye-tracking that can automatically analyze and contextualize gaze in traffic and warn vulnerable road users of danger. This thesis consists of four independently readable and empirical research papers.

The first study examines the effect of drivers’ eye contact on pedestrians’ crossing decisions using an online crowdsourced experiment. It shows that, although a car’s kinematics have a dominant effect, a driver’s eye contact also makes pedestrians feel safer and more likely to cross the road, and that the timing of the driver’s eye contact has an influence as well. The second study benchmarks the accuracies of mobile eye-trackers under static and dynamic conditions, finding that eccentricity worsens accuracy, but dynamicity does not necessarily worsen it. The third study presents a method to objectively detect and operationalize driver-pedestrian eye contact using two synchronized eye-trackers and computer vision, defining eye contact as mutual gaze within a 4° threshold. The fourth study explores the integration of mobile eye-tracking, object detection, and a vision-language model in an attempt to develop a real-time, context-aware safety system that can assess risk in traffic and enhance the situational awareness of road users.

This dissertation concludes that while eye contact is neither as powerful a cue as kinematics nor essential for crossing, it is still a “should-have” in driver-pedestrian interactions as it can increase perceived safety and willingness to cross. This thesis also concludes that certain types of external Human Machine Interfaces (eHMIs) – substitutes for the missing eye contact between pedestrians and automated vehicles – would be beneficial to maintain existing levels of comfort in interactions. Finally, this thesis also highlights the potential of using mobile eye-tracking in combination with computer vision and AI for applications in the traffic, manufacturing, medical, education, and other domains, and recommends topics for further research into eye contact and eye-tracking.

With the advancement of Artificial Intelligence leading to increasingly human-like outputs, assessing a machine’s ability to exhibit human-like intelligence has become more essential than ever. This study aims to investigate how human-like chess players perceive four conditions: one human opponent and three different types of algorithms. One of these algorithms, Maia, has been trained on human data and aims to play the most human-like move. In a custom-designed experiment similar to a Turing test, chess players faced off against Maia, Stockfish and a human without knowing their opponent’s nature. After each game, the chess player assessed how human-like the moves of the opponent were and estimated whether they played against an engine or a human opponent. During the game, participants were asked to think aloud about their next move and react towards the moves of the opponent. Additionally, the gaze of the player was captured with the SR EyeLink Portable Duo at 1000Hz, with the goal of finding differences within the player’s gaze while participants tried to discover the nature of their opponent. Results from the experiment revealed that, based on responses to a subjective questionnaire, the perceived humanness of Maia is statistically similar to a human and different from the other two chess engines. From the analysis of the voice recordings, categories of sentences were identified that could suggest recognition of the opponent, specifically: "expected", "unexpected", "human-like" and "engine-like". From the eye-tracking results, the average fixation duration and pupil diameter changes following the opponent’s move were compared for each condition, but showed no statistical differences between conditions. In summary, Maia was perceived more human-like compared with other chess engines. However, differences in underlying cognitive processes on how the human perceived this difference in a Turing Test experiment were not identified.

This thesis presents the design and evaluation of a comprehensive system for developing voice-based interfaces to support users in supermarkets. These interfaces enable customers to convey their needs across both generic and specific queries. While current state-of-the-art systems like GPTs by OpenAI are easily accessible and adaptable, featuring low-code deployment with options for functional integration, they still face challenges such as increased response times and limitations in strategic control for tailored use-case and cost optimisation. Motivated by the goal of crafting inclusive, personalised, and efficient conversational agents, this study advances on three fronts: 1) a comparative analysis of four popular off-the-shelf speech recognition technologies to identify the most accurate model for different genders (male/female) and languages (English/Dutch); 2) an assessment of the effects of personalised recommendations versus generic responses, using a blindfolded, counterbalanced within-subject experiment; and 3) the development and evaluation of a novel multi-LLM supermarket chatbot framework, comparing its performance with a specialized GPT model powered by the GPT-4 Turbo, using the Artificial Social Agent Questionnaire (ASAQ) in a counterbalanced within-subjects experiment and qualitative participant feedback. Our find-ings reveal that OpenAI’s Whisper leads in speech recognition accuracy across genders and languages, users significantly prefer personalised chatbots over the non-personalised counterparts and that our proposed multi-LLM chatbot architecture outperformed the benchmarked GPT model across all 13 measured criteria, including statistically significant improvements in four key areas: performance, user satisfaction, user-agent partnership, and self-image enhancement. The thesis concludes by presenting a simple method for supermarket robot navigation by mapping the final chatbot response to correct shelf numbers towards which the robot can plan sequential visits. This later enables effective use of low-level perception, motion planning, and control capabilities for product retrieval and collection. We hope this work encourages more efforts into using multiple, specialised smaller models instead of always relying on a single powerful model.

Immersive virtual reality (IVR) with head-mounted displays (HMDs) is expanding in various fields like training, but its effects on cognitive load from visual, auditory, and mental stimuli in virtual environments remain uncertain. This is particularly relevant in neurorehabilitation, where patients may suffer from training in overstimulating environments due to cognitive impairments. This study further explores how low and high levels of visual, auditory, and cognitive demands affect the cognitive load. Twenty-two participants used an HMD for a virtual shopping task, which involved selecting listed products and placing them in a cart, under baseline (the task without additional demands) and two stimulus complexity levels (low and high) for visual, auditory, and mental demands. The study assessed cognitive load using conventional (heart rate, variability, skin conductance, performance, self-reported questionnaire) and underexplored measures (head stillness, hand smoothness, gaze behavior), and explored behavior changes due to stimulus impact. Results showed that visual and auditory stimuli had minimal effects on cognitive load, with only specific measures showing any notable differences from the baseline. Mental stimuli significantly impacted cognitive load, with high mental tasks notably affecting the measures and behavior, whereas low mental tasks showed fewer changes. This research concluded that mental stimuli significantly increased the cognitive load in virtual environments, more than visual or auditory stimuli, suggesting future virtual reality designs should prioritize managing mental load. Furthermore, the study highlights the effectiveness of head stillness and gaze behavior as innovative measures for evaluating cognitive load.

This thesis explores how automated vehicles will interact with pedestrians in the urban environment through augmented reality technology. Nine distinct AR interfaces were designed, developed, and evaluated to assess how different design elements (symbols, text, colour) and distinct mappings of the AR (on the road, on the vehicle, or head-locked) would affect comprehension, and ultimately whether the pedestrian would trust and be convinced to cross in front of an automated vehicle displaying a safe message. Using increasing levels of ecological validity, from an online questionnaire to a CAVE simulator and an AR HMD experiment, the evaluation also explored how different AR anchoring (and mapping) positions affect pedestrians' crossing initiation times and the intuitiveness of the message. The thesis also explores the use of diminished reality (removal of information) to assist pedestrians in occluded scenarios, as well as the utilisation of Large Language Models in evaluating qualitative data in experiments. The outcomes of the thesis are a set of guidelines based on empirical evidence on how to design effective AR interfaces which promote safe and transparent interactions between pedestrians and automated vehicles.

This dissertation explores cyclist-automated vehicle interactions, emphasising developing and integrating human-machine interfaces (HMIs) to enhance cyclist safety and communication. Adopting a cyclist-centric perspective, it recognises cyclists' unique characteristics and communication strategies in shared traffic environments. Using semi-structured interviews, literature reviews, data triangulation, an eye-tracking field experiment, and a cycling simulator study, the research addresses five key research questions, providing qualitative and quantitative insights.

The main contributions of this dissertation include a thorough investigation of cyclists' expectations for future interactions with automated vehicles, highlighting the need for reliable detection by automated vehicles and placing the responsibility for safety on vehicle developers rather than cyclists. The research offers objective data and self-reported insights into cyclist-automated vehicle interactions and evaluates cyclists' ability to visually detect the presence or absence of a driver. Additionally, it introduces 20 scenarios of cyclist-automated vehicle interaction, serving as a resource for safety assessments and HMI research. A comprehensive literature review of existing HMIs for cyclists was conducted, identifying 92 concepts involving vehicles, bicycles, cyclists, and infrastructure.

The dissertation concludes with design recommendations for cyclist-centric HMIs, proposing an omnidirectional on-vehicle external HMI (eHMI) to communicate detection and automated driving mode. This dissertation provides valuable insights for researchers, policymakers, and automated vehicle developers, aiming for the safer, more inclusive, and sustainable urban traffic environments of tomorrow.

Objective: The aim of this study was to investigate the effect of autonomous-vehicle-to-pedestrian (AV2P) communication through augmented reality (AR) interfaces on the road crossing behaviour of pedestrians, and research whether subjective results from a previous Cave Automatic Virtual Environment (CAVE) study replicated in a real world AR experiment.
Background: Previous studies investigating the effects of AV2P communication have mostly been conducted through virtual reality (VR) providing researchers with safe experimentation methods and high experimental control, but also resulting in a common limitation: the lack of ecological validity and realism, thereby affecting participants’ behaviour and causing distractions. This study therefore introduces AR experiments that have been conducted in a real world environment to increase ecological validity.
Methods: An AR experiment was conducted in which 28 participants were situated in the real world with the objective to cross the road. The virtual vehicle, that was projected through a Varjo XR-3 head mounted display, approached from the right at a speed of 30 km/h while 4 interfaces (2x world-locked, head-locked, and vehicle-locked) appeared to communicate the vehicle’s intention towards the participants, in addition to a no-interface baseline. Participants were tasked with indicating when they were willing to cross through the push of a remote button from which their Willingness to cross and Decision certainty could be derived. Subjective data was collected after the trials and after the experiment through interviews and a questionnaire respectively.
Results: Results suggest a positive effect of the AV2P interfaces on the Willingness to cross and Decision certainty, although statistically not significant. In other words, Willingness to cross increases when the vehicle indicates that it will yield, and decreases when the vehicle communicates that it will not yield. Decision certainty also increases when an interface is present compared to the no-interface baseline. Moreover, participants indicated using the interfaces as a tool to validate their own decisions. Compared to the CAVE study, subjective intuitiveness ratings replicate in terms of observing higher intuitiveness of the interfaces than the no-interface baseline. However, the intuitiveness ratings were higher in the CAVE study than the real world AR experiment. Furthermore, the order of the top 3 most preferred interfaces ranking is in the opposite order. Both differences suggest that the increased ecological validity of the real world AR experiment introduces new insights into participants’ perception of interfaces. The Van der Laan acceptance scale shows that participants believe interfaces to be useful and satisfying overall.
Conclusion: The experiments suggest that AV2P interfaces have a positive effect on the crossing behaviour of pedestrians. Furthermore, participants indicate using the interfaces as a tool to validate their own decision, which increases confidence in their decisions. Although results partially replicate a previous virtual environment study, there are differences that suggest that real world AR experiments provide valuable insights into participants’ perception of interfaces in a more realistic experiment.

As technology continues to evolve at a rapid pace, robots are becoming an increasingly common sight in our daily lives.
Robots that work with humans need to adapt to a variety of users and tasks, and learn to optimise their behaviour. For non-specialist users to interact with such robots, the robot's learning process needs to be transparent through its behaviour. Reinforcement Learning (RL) is a promising learning method to achieve this adaptability. However, the behaviour generated by RL is not inherently transparent because of the exploration/exploitation trade-off that is needed to optimise a policy for a specific task.

A RL algorithm is Temporal Difference (TD) learning. In TD learning, the algorithm updates a Q-table to keep track of Q-values. Q-values represent the expected future rewards that the agent (the actor that decides what action to take) can receive by taking a specific action in a certain state. Calculating the Q-values involves a value called the Temporal Difference, which is the difference between the current Q-value with the received reward added and the Q-value for the future state and chosen action.

Emotions are a natural way of communicating intent and situational appraisal for humans. In this study, emotional expressions based on Temporal Differences were implemented as a means to increase the transparency of a robot's learning progress. The effects on the robot's learning progress, learning result, and user experience were analysed.

A between-subject experiment with 61 participants on the following three robot modes was performed: no emotions, simulated emotions, and simulated emotions with matching attribution (see Table \ref{table:robotModes}). The simulated emotions are hope, fear, joy, and distress, which were expressed by a humanoid robot. The robot mode with simulated emotions and matching attributions would explain for what task it was feeling hope or fear. The task was a simple task where a human teacher had to help a humanoid robot to learn to express three different colours based on human commands.

The results demonstrate minimal differences between these three conditions. This means that for simple tasks, emotional expressions grounded in RL do not have a significant effect, and thus do not help nor hurt. The findings are discussed, and it is proposed that emotion simulation is beneficial for tasks that are more complex, afford some robot autonomy, and for which the emotion is informative about how the user should influence the robot's actions to the benefit of the robot's policy.

Background: For rigorous software testing, integration and end-to-end tests are essential to ensure the expected behavior of multiple interacting components of the system. When software is subjected to integration or end-to-end tests, it is often unfeasible to test every code change individually, as the runtime of these tests is usually significantly larger compared to unit tests. For this reason, batches of code changes from multiple authors are often tested simultaneously. Problem: An issue with testing multiple changes simultaneously is that it can be unclear which change form which author caused the failure when tests fail, as all changes from all authors included in the test can be at fault. Design: To solve this, a new automatic fault localization algorithm called GitFL is introduced, which combines state-of-the-art fault localization with version control history information for enhanced performance. GitFL was evaluated on a C++ repository at Adyen where tests are considered to be end-to-end. Findings: It showed that the addition of version control history information significantly increases the performance of fault localization for systems where multiple changes are tested simultaneously. Societal implications: This work provides insights on improved fault localization for these systems, which could enable organizations which develop these systems to speed up their testing and development processes. Originality: This work contributes by focusing on fault localization specifically for systems where multiple changes are tested simultaneously, which was not researched before.

Cardiovascular diseases (CVDs) are a group of disorders of the heart and blood vessels.
CVDs are the leading cause of death worldwide. To diagnose and treat CVDs, clinicians and cardiologists use multiple noninvasive imaging techniques. These scans are used to segment certain structures of the heart. Deep learning-based cardiac segmentation on short-axis cardiac magnetic resonance images (CMRI) has gained popularity over the past few years because of its generalisability and accuracy. This has exponentially reduced contouring times for clinicians. The development of such deep learning techniques has seen a common trend. In order to accommodate learning for larger cardiac datasets, the depth and size of segmentation networks have been increased. Unfortunately, the environmental impact of exploding such networks is not taken into account. One solution to mitigate having computationally expensive networks is to incorporate anatomical knowledge in the form of shape priors. The Gridnet and UNet with a shape prior are computationally efficient networks that are used to evaluate segmentation performance on a large and varied cardiac dataset (Combination of the Automated Cardiac Diagnosis Challenge - ACDC and Multi-Centre, Multi-Vendor and Multi-Disease Cardiac Segmentation challenge - M&M datasets). On average, these networks segment CMRIs with an average dice score of 0.87 and a Hausdorff distance of 11.7mm. In parallel, one of the major issues in cardiac technology is the under-representation of women in cardiac datasets. Purposefully curated cardiac datasets such as ACDC and M&M try and maintain equal representation. In real-world scenarios, this might not always be the case. Clinical trials to collect such data often report female representation as low as 25%. Evaluation of segmentation performance between a balanced and skewed dataset is conducted. This is to address if bias in such cardiac training datasets affects the performance of segmentation networks between male and female test patients.

An event-based camera enables capturing a video at a high temporal resolution, high dynamical range, reduced power consumption and minimal data bandwidth while the camera has minimal physical dimensions compared to a frame-based camera with the same vision properties. The limiting factor, however, of an event-based camera is the spatial resolution which ranges between 40 × 40 and 1280 × 960. To counter this deficiency, a method is researched to super resolve event-based vision in order to enhance spatial resolution. A selection of different neural network types and configurations are researched in a step-by-step fashion. Subsequent experiments tested the selected networks on their ability to process event-based data and extract features from it. Followed by experiments that exploited the limitations of the networks to super resolve at different ratios, lengths of eventstreams and more complex event-based data. Results of various experiments showed that a network configuration that utilizes a transformer architecture was best able to super resolve event-based vision. This type of network leverages the ability to extract features based on dependencies between events which aligns with the characteristics of event- based vision. Based on the obtained results from the exper- iments, a pipeline is proposed to super resolve event-based vision and consists of a combination of a transformer network, multilayer perceptrons and a k-nearest-neighbor algorithm. Using this pipeline, eventstreams can be super resolved in the spatial resolution at a scaling ratio of 4. Visually, these super resolved eventstreams resemble more detailed and enhanced version to the low-resolution input. This proposed pipeline can be considered as a starting point in further research toward the super-resolution of event-based data and thereby contributes to the extension of application possibilities of event-based vision.

Human operators who are tasked with monitoring automation systems may experience a high visual demand to process the information streams from these systems. The visual sampling behavior of human operators can be described using mathematical models. These models can help designers improve environments where multiple signals are present for human operators to monitor, to a configuration that can be processed properly.

This study consisted of two parts. The first part investigated how peripheral vision plays a role in visual sampling behavior and task performance, specifically in the experimental eye-tracking setup presented in Eisma et al. (2018). In this setup, participants were instructed to monitor a bank of six dials, of which each dial pointer had a threshold indicator, and press a response key whenever a dial pointer crossed the threshold indicator. In the second part, the sampling models as presented by Senders (1983) are implemented to predict sampling trajectories. The sampling characteristics that resulted from the
predictions were then evaluated.

The results of the experiments show that peripheral vision plays a role in visual sampling and task performance. More specifically, sampling behavior is more evenly distributed among dials, and task performance is lower when peripheral vision is absent. The main attractor in the peripheral vision is shown to be the pointer speed. Moreover, the learning effect presented in Eisma et al. (2018) is not apparent when peripheral vision is absent.

The results of the predictions showcase the sampling behavior characteristics, some of which show similarities with the results from the experimental data.

Objective: In this thesis, we explore whether augmented and diminished reality interfaces, which, respectively, add and remove information from the environment, improve a pedestrian's feeling of road crossing safety, and how this information should be conveyed to the pedestrian.

Background: Literature shows that view occlusion is a prominent cause in pedestrian collisions. The research focus is currently on vehicle technology and pedestrian warning systems. Whether aiding pedestrians with camera views from unobstructed positions helps to overcome the view occlusion problem is unclear.

Methods: Twenty-eight participants engaged in a virtual reality urban road crossing scenario, in which they took on the role of a pedestrian. The pedestrian was situated on the curb and positioned such that the view on the road was largely obstructed. An autonomous vehicle approached and drove past from the left of the pedestrian. Through a head-mounted display, the participants experienced seven prototypes: baseline (i.e., no display), see-through display, transparent car, and both a head-locked and body-locked display with and without view guidance. The order in which participants encountered the prototypes was determined by a balanced Latin square, and each interface was tested by means of six trials with a non-yielding and a yielding scenario randomly selected such that in total three non-yielding and three yielding scenarios occurred in each block. The participants were instructed to continuously indicate whether they felt safe to cross by pressing a button. The interface's acceptance, workload and preference were measured with questionnaires.

Results: The participants' perceived feeling of safety revealed improved performance for all interfaces compared to the baseline condition. For the baseline condition, in which the vulnerable road user did not have access to occlusion-free information, the perceived feeling of safety was the lowest on average and decreased the earliest in the autonomous vehicle approaching phase, as well as scoring the lowest rating on acceptance. The see-through display and the transparent car interfaces, which used a combination of augmented and diminished reality properties to convey the information in a world-anchored manner to the pedestrians, achieved a higher acceptance and perceived feeling of safety than the head-locked and body-locked display interfaces.

Conclusion: A vulnerable road user's perceived feeling of safety can be increased by means of camera views from unobstructed positions to help overcome the view occlusion problem in common road crossing scenarios. This study's findings suggest a positive effect for diminished reality techniques for pedestrians, and future research could examine this technology further in more demanding scenarios.

Background: Merging on a highway is a complex driving task that requires a lot of interaction with other road users. During these tasks, a driver is required to evaluate gaps in space and time between the themselves and other road users and obstacles in order to arrive at the right moment to merge onto the highway. To improve safety and increase road efficiency, it is necessary to understand the decision process during merging decisions. A way of achieving this, is to understand what visual information humans use during this decision process. This study investigated the relation between gaze location and gap acceptance decisions during highway merges.
Methods: An experiment was performed in which 26 participants monitored an automated vehicle (AV) that was driving on a highway on-ramp. The participants were given the task to train the AV in whether or not to merge in front of an upcoming vehicle that was already driving on the highway. An eye tracker was used to measure gaze data, which was used to find the relation between gaze behaviour and decision outcomes and response times. A mixed-effects logistic model was used for a statistical analysis with decision outcomes as a dependent variable and different gap sizes as predictor variables. A mixed-effects linear model was used to find the relation between response times and dwell times and the different gap sizes and decision outcomes as predictor variables. For both the decision outcome and response time model, dwell time was later included to find the effect on the predictive validity.
Results: The results show that a larger time and distance gap to the upcoming vehicle relate to a higher merging probability. For larger time gaps to the on-ramp, the probability of merging was found to be smaller. It was also found that time gaps to the end of the on-ramp significantly relate to response times, with an increase of 55ms per 1s. Larger time gaps to the upcoming vehicle significantly relates to larger response times, with an increase of 64ms per 1s. No significant relation was found between response time and distance gaps to the upcoming vehicle. The response time was found to be 0.60s longer for rejected gap decisions. The time gap to the end of the on-ramp significantly relates to dwell time, with an increase of 0.56% per 1s. The distance gap to the upcoming vehicle significantly relates to dwell time, with an increase of 0.60% per 10m. The time gap to the upcoming vehicle significantly relates to dwell time, with an increase of 0.52% per 1s. The presented results show as well that a significant relation exists between gaze behaviour and decision outcomes and response times. When analysing decision outcomes and response times, the interaction between dwell time and gap sizes should be taken into account. This improved the predictive validity of the used regression models.
Conclusion: Several pieces of evidence suggest that gaze behaviour assist in understanding the human decision making process during merging. This study can serve as a basis for cognitive models that can investigate how the relation between gaze behaviour and gap sizes, decision outcomes and response times can help to understand and potentially predict gap acceptance decisions.

With the increase in development of self-driving cars, research has been conducted to retain humanized interaction between cars and other road users, such as pedestrians. One way to retain this type of interaction is through the use of external Human-Machine Interfaces (eHMIs). This project aims to contribute to this field of research by exploring the case of eHMI-equipped, self-driving cars yielding for a crossing pedestrian. From literature it is known that light- and text-based signals are both promising ways of utilizing an eHMI. Due to the often simplified nature of these models, the goal of this project was to investigate if their findings hold up when increasing the cognitive load of the pedestrian. An eye-tracking experiment was conducted where two promising eHMI signal types (i.e., flashing lights and message) were tested in a realistic scenario, where the participant took the role of pedestrian. In the experiment, the cognitive load of the participants was varied, by applying different sizes of gaze contingent windows to the stimuli.

A range of 63 different trials of 10s each were shown to 23 participants. This set of trials included 7 different presets, aimed to test the effect of the signal type and the gaze window size independently. The participants' task during the experiment was to indicate when they deemed it safe to cross the road, by pressing the space bar. Their gaze was measured with an eye-tracker, after which it was transformed into three different metrics: saccade count, saccade amplitude and fixation duration. Additionally, a novel metric, the dispersion of the grouped gaze data, was introduced and explored. Finally, a questionnaire was conducted, investigating the self-reported clarity of the different signal types. Together with the reaction time, these metrics aimed to answer the following research question: What influence does raising the cognitive load for crossing pedestrians have on the effectiveness of text-based eHMIs as opposed to light-based eHMIs and no eHMI?

The results show that, as has been previously established, light- and text-based signals showed very similar response times and feature similar gaze characteristics when compared to each other. Both outperformed the condition where no signal was shown. Raising the cognitive load shows a decrease in saccade count and saccade amplitude, paired with a higher fixation duration. However, no proof could be found that raising the cognitive load has an influence on the effectiveness on any of the different signal types, meaning that either, the cognitive load was not raised by enough, or there is no actual effect. The dispersion showed that for the light-based signal, the focus on the stopping car is lost the quickest after the signal was shown.

The results of this study may help in explaining how pedestrians base their traffic decision on the type of signals being shown. This may also serve as a basis to further explore the possible effects of cognitive load on the effectiveness of these types of signals.

The main task during operating an automotive vehicle is driving. Nowadays, distractions form a potential risk of claiming the workload necessary for the driving task. Interacting with the User Interface (UI) of the vehicle can be such a distraction. Predicting the next action on the UI can help decrease the risk of distractions. To predict the next user action, knowledge about the UI interaction behavior is necessary.
In this work, a descriptive analysis is conducted on a large naturlistic dataset which has not been matched in size by other related work in the field of automotive Human Machine Interface (HMI) research. The analysis is conducted to gain insight into the interaction behavior of the drivers. This behavior is related to several driving task conditions such as occupancy, speed and drive mode.
The results of the descriptive analysis are used to implement several prediction models to predict the next user action. Among the models, the Long Short-Term Memory (LSTM)-network achieved a validation accuracy of 87%. This work shows the potential for analysing UI interaction behavior and leveraging this for prediction purposes. It could help in forming a foundation for future adaptive UI work.

Vulnerable road users account for more than 50% of traffic fatalities, and among these, pedestrians are the most susceptible to fatalities due to their distraction and misperception of other road users. To mitigate their plight, systems that warn drivers and pedestrians in case of a possible collision have been developed. Among systems that focus on the pedestrian's perspective, existing concepts are capable of predicting collisions but lack elements that monitor the visual attention of pedestrians. We address this gap by developing a gaze-based pedestrian warning system based on the Tobii Pro Glasses 2, a head-mounted eye-tracker. The system consists of: (1) a custom trained fast neural network (YOLO v4) on the KITTI object detection dataset that processes the video feed of the eye-tracker to detect approaching vehicles and (2) a module that uses the pedestrian's gaze to identify whether their attention falls on the closest moving vehicle that is approaching the pedestrian, both in real-time. If the pedestrian does not look at the approaching vehicle, they are given an auditory alert that warns them of a possible collision. In a pilot study conducted on a busy road in an urban environment, the system was evaluated under different pedestrian walking speeds and gaze behaviours to test the algorithm's robustness. The pilot study revealed that our system alerted the inattentive pedestrian with an accuracy of 67%. The mean vehicle detection accuracy and a mean moving vehicle identification accuracy from the pilot were 93% and 60%, respectively, a promising result given the use of only a mono camera. Despite the use of computer vision techniques, the system worked at an inference speed of 50 FPS due to the multi-processing capabilities of our algorithm. Our efforts are a first step in developing pedestrian warning systems based on eye-tracking technology to improve road safety in the future. The algorithm (Python-based) code used for this work has been made publicly available.

One of the current challenges withholding personalised lane-level driving advice is the inaccuracy and error of GPS signal from commonly used navigation devices and mobile phones. These GPS signals have an uncertainty margin up to several meters, therefore potentially indicating the vehicle location on a different lane than the actual lane it would be in. This unreliability therefore currently makes it impossible to accurately recognise lane changes from solely this data. This study looks into the recognition of lane changes from only Floating Car Data by the use of a Random Forest algorithm. In order to find the ground truth, a trajectory reconstruction algorithm is implemented, which uses the matching of trajectories with loop detector passages in order to find the lane a vehicle is in at each loop detector location. This information is then used to know whether, for each vehicle, a lane change is made on the road section in-between two consecutive loop detector locations. By training the model on this data, it was found that when using solely Floating Car Data, lane changes can be recognised with an accuracy of up to 64%. Indicators for lane change were found to be the lateral distance of a vehicle to the middle of the road, as well as the heading of the vehicle. The study additionally looks into a rule based method of lane change recognition, which is compared with the Random Forest model.

Although the capabilities of automated driving systems (ADS) are growing, the human operator remains in charge of driving the vehicle outside the operational design domain and after requests to intervene. The vehicle sensors required for ADS and new regulation for driver availability monitoring systems stimulate the development of systems that can monitor human driving ability. Such systems could allow for ADS to track possible driving skill degradation, monitor driver impairment, or adapt to the driver’s needs. However, knowledge is required about how to assess human driving ability effectively. This work proposes a novel method to capture driving skill and style based on curve driving, straight-road driving, and driving through road narrowings. In addition to conventional measures, the study introduces new measures, including the relationship between steering angle and eye movements. Employing a high-fidelity simulator, we compared the vehicle-control-related measures (i.e., driving skill measures), trajectory-planning and speed-related measures (i.e., driving style measures), and eye movements of sixteen inexperienced and thirteen experienced drivers. It was examined which of these measures are valid, i.e., allow a statistical discrimination between experienced and inexperienced drivers. The study was complemented with two expert drivers serving as a benchmark. The results showed that the experienced drivers adopted a more abrupt braking style when approaching curves and a higher speed through the different track sections than the inexperienced drivers. No statistically significant differences were observed between the skill measures of the experienced and inexperienced drivers. An analysis of lead times obtained through a cross-correlation between horizontal gaze angle and steering angle showed that eye movements generally preceded steering movements. However, differences in lead times between experienced and inexperienced drivers were not statistically significant, which may have been caused by eye-tracking measurement inaccuracies, the layout of the driving task and strong intra-individual variability in looking behavior. The eye-tracking results of the road-narrowings showed that the experienced and inexperienced drivers reduced their vertical gaze dispersion, while only the experienced drivers statistically significantly reduced their horizontal gaze dispersion compared to the straights. In other words, the experienced drivers showed increased horizontal gaze tunneling from the straights to the narrowings, while the inexperienced drivers only showed vertical gaze tunneling. The results of the expert drivers showed consistent higher speed, lower control activity, and more of a racing line through the curves than the experienced and inexperienced drivers. Furthermore, the results showed that the expert drivers adopted a more variable horizontal gaze strategy between different curves. Overall, these results indicate that driving style and eye-movement measures, but not driving skill measures, allow differentiating between experienced and inexperienced drivers using a driving simulator. These findings may be explained by the fact that driving is a self-paced task, i.e., more competent drivers increase their own task demands by driving faster. Future research could examine how strongly scores the present driving and eye-movement measures correlate with drivers’ take-over quality in automated driving scenarios.