Current visuomotor manipulators jointly train their perception-planning-action components simultaneously using an end-to-end framework to avoid hand-engineering components. Despite this, methods for human-to-robot object handover tasks require a perception component that segments the hand from the object, which can introduce error propagation. For this reason, this study investigates the applicability of an end-to-end framework that eliminates the need for hand-object segmentation in a simulated human-to-robot object handover task using HandoverSim.

To address this, a behavior cloning agent is used to convert camera input into RGB-D voxel space and output discretized 6-DoF manipulation to directly discover features for the handover task. This study introduces a framework that combines the behavior cloning agent with the HandoverSim, which allows experimenting with various training configurations. These configurations consist of experiments with: 1) expert demonstration data; 2) optimal camera setup; 3) handover objects; and 4) voxel-based RGB augmentation techniques.

The trained model is evaluated on its generalization to diverse handover conditions in the HandoverSim Benchmark. The results demonstrate that the behavior cloning agent can learn features for the handover task without requiring a perception component. The model learns the grasp-object relation whilst minimizing contact with the hand. Despite this, performance is limited by sparse training data and grasping accuracy.

With mobile robotics being applied for more and more complex applications, their autonomy should be preserved. While a lot of research is performed into the direction of failure prediction for autonomous processes or systems, the field of mobile robots has received less attention. Proactive failure prediction for mobile robots is a useful tool to prevent unwanted downtime and undesired damages. This work attempts to fill this research gap by showing the applicability of anomaly detection methods for failure prediction in the field of mobile robots. Specifically, we employ an unsupervised Variational Autoencoder to predict failures in the operational data from the Discovery Collector, a manure cleaning robot developed by Lely Industries. We elaborately showcase the feature engineering steps which yield the best performance, provide the performance of three general datasets, and state promising next steps for root cause classification which is enabled by accurate failure prediction. All in all, our work shows that the use of feature offsets, calculated from desired values compared to actual values, enhances the model performance tremendously. The provided datasets showcase F1-scores ranging from 0.64-0.76, showing the proposed solution is able to solve the failure prediction problem in the field of mobile robots, while highlighting the encountered limitations for future improvement.

Large Language Models (LLMs) possess significant semantic knowledge about the world, making them valuable for high level control for robots through Vision-Language-Action (VLA) models. These models integrate an LLM to deduce semantic knowledge from a robot’s vision and natural language inputs, facilitating real-world actions. Despite their potential, VLA models are a relatively new research area, with applications mostly limited to simulations or household tasks and insufficient validation in broader contexts. This study aims to develop a LLM-Enhanced Robotic Affordance and Control System (LERACS) for robotic manipulation in applied cases such as the management and maintenance of electrical grid infrastructure. LERACS is designed to visually ground manipulable objects and decompose tasks based on user instructions within a human-robot interaction chat interface, using ChatGPT. A system validation and an AI user experiment were conducted to evaluate its effectiveness in interpreting and performing actions based on pre-made and synthetically generated user instructions. These assessed LERACS’ performance across various settings and instructions. Results indicate high success rates in environmental interpretation and task execution, with robust labeling accuracy, especially in complex settings. Feedback from the AI user experiment highlighted LERACS’ adaptability, identified areas for improvement, and demonstrated its practical utility across diverse settings and task complexities.

Surgical navigation involves transferring preoperative imaging data, along with preplanned information, onto the patient in the operating theater without using constant radiation. This technique has proven effective and is widely adopted across various surgical specialties. Research has shown a consistent trend in surgical robotics, with numerous initiatives using this technology to navigate the robot’s end-effector within the patient’s anatomy. For this purpose, commercially available surgical navigation systems are often employed. However, these systems, which are primarily dominated by optical tracking, are not necessarily suited for robotic systems and exhibit limitations such as low update frequency and line-of-sight issues. Additionally, performance reporting in current surgical robotic research is highly inconsistent, and clear guidelines are lacking. This research aims to develop a surgical robotic navigation system to work towards establishing a performance benchmark and systematically assess various error components as a first step toward guiding the field of surgical robotic navigation. To this end, two systems, the Haply System and the Dual-Robot System, have been developed and evaluated for technical accuracy and registration accuracy in both static and dynamic environments. Furthermore, sensor fusion methods have been explored to enhance performance in the Haply system. The results and analysis indicate that the Dual-Robot System is the most accurate in dynamic navigation and presents a viable alternative to optical tracking systems in terms of performance. However, its clinical adoptability remains questionable.

With the advancement of Artificial Intelligence leading to increasingly human-like outputs, assessing a machine’s ability to exhibit human-like intelligence has become more essential than ever. This study aims to investigate how human-like chess players perceive four conditions: one human opponent and three different types of algorithms. One of these algorithms, Maia, has been trained on human data and aims to play the most human-like move. In a custom-designed experiment similar to a Turing test, chess players faced off against Maia, Stockfish and a human without knowing their opponent’s nature. After each game, the chess player assessed how human-like the moves of the opponent were and estimated whether they played against an engine or a human opponent. During the game, participants were asked to think aloud about their next move and react towards the moves of the opponent. Additionally, the gaze of the player was captured with the SR EyeLink Portable Duo at 1000Hz, with the goal of finding differences within the player’s gaze while participants tried to discover the nature of their opponent. Results from the experiment revealed that, based on responses to a subjective questionnaire, the perceived humanness of Maia is statistically similar to a human and different from the other two chess engines. From the analysis of the voice recordings, categories of sentences were identified that could suggest recognition of the opponent, specifically: "expected", "unexpected", "human-like" and "engine-like". From the eye-tracking results, the average fixation duration and pupil diameter changes following the opponent’s move were compared for each condition, but showed no statistical differences between conditions. In summary, Maia was perceived more human-like compared with other chess engines. However, differences in underlying cognitive processes on how the human perceived this difference in a Turing Test experiment were not identified.

As technology continues to evolve at a rapid pace, robots are becoming an increasingly common sight in our daily lives.
Robots that work with humans need to adapt to a variety of users and tasks, and learn to optimise their behaviour. For non-specialist users to interact with such robots, the robot's learning process needs to be transparent through its behaviour. Reinforcement Learning (RL) is a promising learning method to achieve this adaptability. However, the behaviour generated by RL is not inherently transparent because of the exploration/exploitation trade-off that is needed to optimise a policy for a specific task.

A RL algorithm is Temporal Difference (TD) learning. In TD learning, the algorithm updates a Q-table to keep track of Q-values. Q-values represent the expected future rewards that the agent (the actor that decides what action to take) can receive by taking a specific action in a certain state. Calculating the Q-values involves a value called the Temporal Difference, which is the difference between the current Q-value with the received reward added and the Q-value for the future state and chosen action.

Emotions are a natural way of communicating intent and situational appraisal for humans. In this study, emotional expressions based on Temporal Differences were implemented as a means to increase the transparency of a robot's learning progress. The effects on the robot's learning progress, learning result, and user experience were analysed.

A between-subject experiment with 61 participants on the following three robot modes was performed: no emotions, simulated emotions, and simulated emotions with matching attribution (see Table \ref{table:robotModes}). The simulated emotions are hope, fear, joy, and distress, which were expressed by a humanoid robot. The robot mode with simulated emotions and matching attributions would explain for what task it was feeling hope or fear. The task was a simple task where a human teacher had to help a humanoid robot to learn to express three different colours based on human commands.

The results demonstrate minimal differences between these three conditions. This means that for simple tasks, emotional expressions grounded in RL do not have a significant effect, and thus do not help nor hurt. The findings are discussed, and it is proposed that emotion simulation is beneficial for tasks that are more complex, afford some robot autonomy, and for which the emotion is informative about how the user should influence the robot's actions to the benefit of the robot's policy.

This document lays out the design and development of a specialized device aimed at monitoring the eye movements of premature babies, which creates an opportunity to create better sleep-monitoring systems.

Monitoring sleep has great significance, especially for premature babies, because it allows for the adaptation of care to accommodate for better and more sleep. Different sleep stages can be identified, but the identification does differ for adults compared to premature babies. Different characteristics including brain activity, heart rate, respiration, eye movement and face and body movements are linked to those sleep stages.

Eye movement is a characteristic that could be used more optimally for the creation of a sleep-monitoring system. Eye movement can be tracked using different methods, but electrooculography shows the greatest promise. For this specific method, different options exist concerning electrode placement and materials. The use of this system for premature babies during sleep also creates limitations related to the fragility of the skin of prematurely born babies. Furthermore, these babies normally lie in the Neonatal Intensive Care Unit, creating additional context-specific considerations.

Different research methods were used, resulting in a complete picture of the steps taken after premature birth occurs. These steps are portrayed in a scenario. The requirements and wishes drawn from these are used to develop ideas. During the development, a choice was made to create two parts for the device, namely the shell and the electronics module.

In the final design, the shell contains specialized electrode places, named snap-rings, as well as features that create adaptability and additional details to accommodate the use of multiple devices that are already being used on premature babies. The electronics module includes an electrode configuration that can be placed into the snap-rings in the shell. The electrode protrudes the snap-ring and is pushed onto the skin. The force needed to place the electrode is decreased by using a material with a low Young's modulus, ensuring contact between the electrode and the skin without causing damage to it. This novel configuration enables the use of dry electrodes, eliminating the need for gels and adhesives.

Some recommendations are given, which concern further the development of the device, the eventual certification and the embedding of the device within a system that monitors sleep.

Overall, a complete overview of the concept design of a device that can monitor eye movement of premature babies is given, which creates an opportunity for the improvement of sleep monitoring systems for these babies.

With the introduction of automated driving systems come benefits such as the improvement of traffic safety. However, with an increasing level of automation in vehicles also comes an increase in interaction with in-vehicle technology by drivers while they are meant to supervise the automated driving systems. Due to more interaction with in-vehicle technology and a vigilance decrement of the driver in Level 2 driving, an increase in reaction time of the driver is seen when intervention is needed by the means of a take over request. This delay in reaction by the driver opposes the benefit of the introduction of automated driving systems and causes hazardous situations. To try and circumvent the effect of vigilance decrement, this paper attempts to demonstrate the reduction of noticing time of the Hands-On-Wheel warning message for drivers of Level 2 vehicles while interacting with in-vehicle technology through the implementation of a gaze-contingent interface. The results of this experiment indicate a 79.3% lower noticing time of the Hands-On-Wheel warning message when the stimulus is placed in a gazecontingent manner, while the participants engage in secondary tasks on the in-vehicle technology. The placement of the stimulus on the head unit when the participant is already looking at it reduces the primary task load of touching the steering wheel and causes for the stimulus to be seen quicker as compared to a static interface. However, the performance of the secondary task seems to decrease when using a gaze-contingent interface. This is due to the intrusive nature of the placement of the stimulus, which demands the driver to store information regarding the secondary task in their working memory while they attend to the primary task. Despite the decline in secondary task performance, the reduction of noticing times of time critical messages when placed in a gaze-contingent manner could be beneficial to the safety of autonomous driving functions where the driver has a vigilance task and is engaging in secondary tasks.

Human operators who are tasked with monitoring automation systems may experience a high visual demand to process the information streams from these systems. The visual sampling behavior of human operators can be described using mathematical models. These models can help designers improve environments where multiple signals are present for human operators to monitor, to a configuration that can be processed properly.

This study consisted of two parts. The first part investigated how peripheral vision plays a role in visual sampling behavior and task performance, specifically in the experimental eye-tracking setup presented in Eisma et al. (2018). In this setup, participants were instructed to monitor a bank of six dials, of which each dial pointer had a threshold indicator, and press a response key whenever a dial pointer crossed the threshold indicator. In the second part, the sampling models as presented by Senders (1983) are implemented to predict sampling trajectories. The sampling characteristics that resulted from the
predictions were then evaluated.

The results of the experiments show that peripheral vision plays a role in visual sampling and task performance. More specifically, sampling behavior is more evenly distributed among dials, and task performance is lower when peripheral vision is absent. The main attractor in the peripheral vision is shown to be the pointer speed. Moreover, the learning effect presented in Eisma et al. (2018) is not apparent when peripheral vision is absent.

The results of the predictions showcase the sampling behavior characteristics, some of which show similarities with the results from the experimental data.

With the increase in development of self-driving cars, research has been conducted to retain humanized interaction between cars and other road users, such as pedestrians. One way to retain this type of interaction is through the use of external Human-Machine Interfaces (eHMIs). This project aims to contribute to this field of research by exploring the case of eHMI-equipped, self-driving cars yielding for a crossing pedestrian. From literature it is known that light- and text-based signals are both promising ways of utilizing an eHMI. Due to the often simplified nature of these models, the goal of this project was to investigate if their findings hold up when increasing the cognitive load of the pedestrian. An eye-tracking experiment was conducted where two promising eHMI signal types (i.e., flashing lights and message) were tested in a realistic scenario, where the participant took the role of pedestrian. In the experiment, the cognitive load of the participants was varied, by applying different sizes of gaze contingent windows to the stimuli.

A range of 63 different trials of 10s each were shown to 23 participants. This set of trials included 7 different presets, aimed to test the effect of the signal type and the gaze window size independently. The participants' task during the experiment was to indicate when they deemed it safe to cross the road, by pressing the space bar. Their gaze was measured with an eye-tracker, after which it was transformed into three different metrics: saccade count, saccade amplitude and fixation duration. Additionally, a novel metric, the dispersion of the grouped gaze data, was introduced and explored. Finally, a questionnaire was conducted, investigating the self-reported clarity of the different signal types. Together with the reaction time, these metrics aimed to answer the following research question: What influence does raising the cognitive load for crossing pedestrians have on the effectiveness of text-based eHMIs as opposed to light-based eHMIs and no eHMI?

The results show that, as has been previously established, light- and text-based signals showed very similar response times and feature similar gaze characteristics when compared to each other. Both outperformed the condition where no signal was shown. Raising the cognitive load shows a decrease in saccade count and saccade amplitude, paired with a higher fixation duration. However, no proof could be found that raising the cognitive load has an influence on the effectiveness on any of the different signal types, meaning that either, the cognitive load was not raised by enough, or there is no actual effect. The dispersion showed that for the light-based signal, the focus on the stopping car is lost the quickest after the signal was shown.

The results of this study may help in explaining how pedestrians base their traffic decision on the type of signals being shown. This may also serve as a basis to further explore the possible effects of cognitive load on the effectiveness of these types of signals.

The main task during operating an automotive vehicle is driving. Nowadays, distractions form a potential risk of claiming the workload necessary for the driving task. Interacting with the User Interface (UI) of the vehicle can be such a distraction. Predicting the next action on the UI can help decrease the risk of distractions. To predict the next user action, knowledge about the UI interaction behavior is necessary.
In this work, a descriptive analysis is conducted on a large naturlistic dataset which has not been matched in size by other related work in the field of automotive Human Machine Interface (HMI) research. The analysis is conducted to gain insight into the interaction behavior of the drivers. This behavior is related to several driving task conditions such as occupancy, speed and drive mode.
The results of the descriptive analysis are used to implement several prediction models to predict the next user action. Among the models, the Long Short-Term Memory (LSTM)-network achieved a validation accuracy of 87%. This work shows the potential for analysing UI interaction behavior and leveraging this for prediction purposes. It could help in forming a foundation for future adaptive UI work.

Vibrotactile wearable devices are a non-intrusive and inexpensive means to provide haptic feedback directly on the user’s skin. These devices utilize one or multiple vibrotactile actuators to generate vibrations across the skin and into the tissue. Combining these vibrations in amplitude can create the illusion of a funneled sensation on the skin at another location than at the actual sites of stimulation. This allows for the placement of virtual actuators on the skin, such that fewer actuators need to be deployed. However, the illusion does not take into account that the waves originating from the actuator attenuate and disperse due to the viscoelastic properties of the skin. We hypothesize that this diffusion of the elastic energy in the skin is affecting the perception of this illusion. Therefore, if we correct for the wave propagation speed, and temporally focus the stimulation, we hypothesized that the specificity of the stimulation on the skin could be drastically improved. In this paper, a novel technique, which is named the inverse filter technique, was introduced that enables to focus the amplitude, frequency and phase of vibrations to one location while cancelling them at the remaining nearby positions. We developed a wearable device for the volar surface of the forearm on which we could independently control arbitrary waveforms at any position between a set of four physical actuators. A human-subject study found that the performance in terms of localization confidence was improved significantly, whereas the precision and accuracy of the task did not improve compared to when we did not correct for the wave attenuation and dispersion. These results show that focusing waves towards a target location has a direct influence on our confidence of localizing vibrotactile stimuli on the arm. Therefore, we anticipate that our findings can benefit industries interested in including localized vibrotactile feedback on the human body surface.

Although the capabilities of automated driving systems (ADS) are growing, the human operator remains in charge of driving the vehicle outside the operational design domain and after requests to intervene. The vehicle sensors required for ADS and new regulation for driver availability monitoring systems stimulate the development of systems that can monitor human driving ability. Such systems could allow for ADS to track possible driving skill degradation, monitor driver impairment, or adapt to the driver’s needs. However, knowledge is required about how to assess human driving ability effectively. This work proposes a novel method to capture driving skill and style based on curve driving, straight-road driving, and driving through road narrowings. In addition to conventional measures, the study introduces new measures, including the relationship between steering angle and eye movements. Employing a high-fidelity simulator, we compared the vehicle-control-related measures (i.e., driving skill measures), trajectory-planning and speed-related measures (i.e., driving style measures), and eye movements of sixteen inexperienced and thirteen experienced drivers. It was examined which of these measures are valid, i.e., allow a statistical discrimination between experienced and inexperienced drivers. The study was complemented with two expert drivers serving as a benchmark. The results showed that the experienced drivers adopted a more abrupt braking style when approaching curves and a higher speed through the different track sections than the inexperienced drivers. No statistically significant differences were observed between the skill measures of the experienced and inexperienced drivers. An analysis of lead times obtained through a cross-correlation between horizontal gaze angle and steering angle showed that eye movements generally preceded steering movements. However, differences in lead times between experienced and inexperienced drivers were not statistically significant, which may have been caused by eye-tracking measurement inaccuracies, the layout of the driving task and strong intra-individual variability in looking behavior. The eye-tracking results of the road-narrowings showed that the experienced and inexperienced drivers reduced their vertical gaze dispersion, while only the experienced drivers statistically significantly reduced their horizontal gaze dispersion compared to the straights. In other words, the experienced drivers showed increased horizontal gaze tunneling from the straights to the narrowings, while the inexperienced drivers only showed vertical gaze tunneling. The results of the expert drivers showed consistent higher speed, lower control activity, and more of a racing line through the curves than the experienced and inexperienced drivers. Furthermore, the results showed that the expert drivers adopted a more variable horizontal gaze strategy between different curves. Overall, these results indicate that driving style and eye-movement measures, but not driving skill measures, allow differentiating between experienced and inexperienced drivers using a driving simulator. These findings may be explained by the fact that driving is a self-paced task, i.e., more competent drivers increase their own task demands by driving faster. Future research could examine how strongly scores the present driving and eye-movement measures correlate with drivers’ take-over quality in automated driving scenarios.

Abstract— Visual occlusion can cause object detection and classification systems to fail due to swift movement or a soiled, moist or dusty environment. Hence, this asks for a ’blind’ collision detection and classification method. This paper presents a novel blind collision detection and classification scheme for a
heavyweight, low-speed, wheeled mobile robot. By recognizing the target signal pattern in phase current data, the robot can detect whether the wheels are blocked caused by overload (G1). After this, using acceleration and phase current data, the robot will be able to differentiate between a collision against a wall
or a collision with an object between the wall, or no collision of interest (G2). Additionally, an algorithm based on the pitch angle can detect if the robot slides on top of a soft object (G3). Three different algorithms are developed that address these goals. Finally, the developed algorithms are validated in
dynamic and soiled environment (G4). The first two goals are reached by training machine learning classifiers that identify the signal pattern of its target event and place uneventful data in a separate class (open-world classification). During training the classifiers Random Forest, Multi-layered Perceptron, Decision
Tree and Logistic Regression are compared, and the two best perfoming classifiers are subsequently tested for a time-series open-world classification task. The third goal is reached with the development of a heuristic algorithm based on the running mean of the pitch angle.
Experiments were performed in a controlled environment, creating collisions with varying floor conditions, robot weight and collision objects. The best performing algorithm for blind collision overload detection has achieved 98.6% detection accuracy. The object classification algorithm can differentiate
between two types of soft objects, a wall or no event with an accuracy of 93.3%. The algorithm based on the deviating pitch angle can detect events with 100% detection rate and 0% false positive rate. Detailed implementation schemes are provided for real-time implementation, illustrating a robust framework in soiled environments. The proposed solutions can be used to improve collision detection on blind mobile robots, as well as mapping the environment using the object classification model.

Whilst the extraction of teeth (exodontia) remains one of the oldest and most performed surgeries on earth, very little is understood about the procedure itself. Especially in the area of the required movements, torques and forces to remove specific teeth and how these interact with existing tissue. This knowledge gap has been hypothesized to contribute to an increasing referral rate towards Oral and Maxillofacial Surgery (OMFS) practices in the Netherlands for simple extractions due to low confidence in young dentists. The objective of this project is to apply techniques used in imitation-learning in the field of robotics to deconstruct complex movement into smaller fundamental building blocks called movement primitives (MPs) to deepen the understanding of exodontia. To achieve this an existing dataset was used consisting of high resolution force-, torque-, position- and rotation-data of extractions. This dataset was collected using a measurement setup consisting of fresh frozen cadaver material, a force torque sensor and a robotic arm in gravity compensation mode. In this paper a novel iterative two staged method for identifying movement primitives is introduced and employed to extract movement primitive information from these extractions in the dataset.

Different types of sensors that monitor the driver, the vehicle, and the surroundings are increasingly being implemented in vehicles. These developments are relevant to formal driving test organizations for the use of sensors in the assessment of driving behaviour. However, no guidelines exist about the exact use of sensor-generated data for driving assessment purposes, such as the driving exam. Relatively low-cost and easily implementable sensors such as accelerometers (g-force sensors) could be used to assess driving behaviour during the driving exam to distinguish between desired and undesired driving behaviour. Earlier studies have already investigated the use of accelerometers to distinguish between driving styles using threshold values, but do not agree on the most optimal threshold to do this.

In this study, thresholds were created based on scripted driving exams driven by professional driving examiners, who portrayed driving styles commonly observed at the driving exam. These rides were driven over a period of three weeks at the Dutch driving license organisation (CBR) in Leusden, The Netherlands, where scripted exam rides are part of the training for new examiners. Using the accelerometer of an iPhone X, the accelerations generated during 21 rides were measured supported by a dashcam recording the road ahead. The experimenter drove along all rides and registered driving events, including turns, speed increases/decreases and lane changes to relate the measured accelerations to these specific events. The rides included in the dataset were divided into four different driving styles: aggressive, desired, overcautious and negligent, from which the first three were analysed due to their direct link with accelerations. The obtained acceleration data was analysed using 63 initial features which describe the amplitude of the acceleration signal through the various events, speed zones and acceleration directions.

It was found that accelerations differ significantly between aggressive and desired driving styles, whilst the difference between overcautious and desired driving is less apparent. The thresholds determined in this study were able to classify the created dataset between desired and undesired driving behaviour with accuracies varying from 50% to 83% (M = 69.1, SD = 11.4).

The results show that using acceleration data obtained during the driving exam can aid the examiner during the driving exam by giving insight into how well a candidate performed in terms of smooth driving through traffic. Concomitantly, a recent report from the Dutch government (Roemer, 2020) addressed that the CBR should investigate the possibilities for using sensor-collected driving data to give more insight to a driving exam candidate as to how to improve his/her driving behaviour after failing the driving exam, giving further rise to researching the use of driving data for practical use.

Mobile soft robots show great potential for exploration of unknown and hard to navigate environments. Sadly, most of these robots are currently being held back by their power sources and control systems. These components, which are usually quite heavy and bulky, need to be integrated before mobile soft robots can cut their tethers and explore the world autonomously. Soft robots should be mindful of their energy consumption to minimize the requirements on the power supply. However, the efficiency of their actuators is still a not well understood area of research. We attempt to pave the road for future research into this subject by developing a testing protocol based on a simplified analytical model. We built an experimental setup and investigated the efficiency of soft pneumatic extension actuators. We found that actuator designs which reduce axial stiffness produced higher efficiency. Additionally, we found that efficiency increases with load, until it is limited by the buckling load. Unfortunately, these two conditions seem to conflict with each other since a lower axial stiffness also reduced the buckling load. Future actuator designs should therefore try to combine a low axial stiffness with high load bearing capabilities. Also, this research should be extended to different classes of soft actuators as well as investigating non-ideal circumstances. We believe this research aids in the general understanding of efficiency of soft actuators and act as a stepping stone for future research.

Background: Automated vehicles are promoted as a safety improvement, but they may also bring a new problem of ‘out of the loop’ errors. Automation support in the form of gaze-contingent feedback might be the solution for these errors. Using the SEEV model, operator’s attention allocation can be predicted and using live gaze data, the driver’s attention guided towards areas of interest.
Methods: An experiment was designed in which twenty participants monitored an automated vehicle and performed a secondary task: either indicating cars driving next to them or monitoring dial crossings. Using an Eye tracker, gaze data was measured and compared to a predictive attention allocation model, gaze-contingent visual feedback was provided when deviating to much from the SEEV prediction. Participants performed each task with and without gaze-contingent feedback in three different driving situations.
Results: The results showed that participants were more actively allocating their attention over all areas of interest with the support of gaze-contingent feedback. They performed more saccades and fixations distributed over the monitor. The fit of the observed percentage dwell time (PDT) on the predicted PDT also improved using gaze- contingent feedback. The task performance of the dial crossing secondary task decreased significantly with feedback on, while the task performance of the hazard perception task did not change.
Conclusion: Attention allocation in complex environments can successfully be predicted using the SEEV model, and implemented in a gaze-contingent feedback model. Attention was divided better and more in line with the predicted PDT with gaze-contingent feedback on, but the current implementation did not improve the secondary task performance.

Problem statement. The introduction of automated vehicles (AVs) changes the role of the driver and may cause a lack of social interaction with pedestrians. This study proposes a concept where the AV is manoeuvre-based controlled via eye gaze, and the AV driver’s gaze is visualised for the driver and pedestrians. However, it was unknown if gaze-based AV control is a viable concept and how the AV’s yielding behaviour should depend on the eye driver’s gaze. Method. A two-agent virtual-reality-based experiment was conducted using two Varjo VR2-PRO head-mounted displays (HMDs). Seventeen pairs of participants (a pedestrian and a driver) each interacted in a road crossing scenario. The pedestrians’ task was to hold a button when they felt safe to cross the road, and the drivers’ task was to direct their gaze according to the instructions. Each session consisted of three blocks of 16 trials: the baseline block, in which the AV driver did not communicate with the pedestrian, and two other blocks in which the driver’s gaze was visualised, namely “gaze at the pedestrian to yield” (GTY) and “look away to yield” (LATY). The effectiveness of the interaction was examined using the pedestrians’ button presses. Acceptance and preference were measured using questionnaires. Results. Pedestrians showed the highest crossing performance and acceptance in the GTY mapping, followed by the LATY mapping and the baseline. The eye gaze visualisation caused pedestrians to spend more time looking at the AV; this effect was particularly dominant when the driver looked at the pedestrian. Conclusion. Gaze visualisation in combination with GTY mapping has the potential to be used as a communication tool for AVs at intersections until full automation of driving (SAE level 5) is technically feasible.