Eye tracking is a cornerstone technology for next-generation human-computer interaction, particularly in Extended Reality (XR), and other healthcare applications. However, traditional frame-based eye tracking systems are constrained by latency, power consumption, and motion blur. Event cameras offer a promising alternative with their high temporal resolution, high dynamic range and low data redundancy, but existing event-based methods often struggle to balance tracking accuracy, computational efficiency, and robustness, especially on resource-constrained mobile hardware.

This thesis addresses these challenges by proposing a novel, purely event-based eye tracking pipeline designed for high-frequency performance and robust accuracy within a strict computational budget. The pipeline accepts only event streams and estimates the pupil region in the field of view. The core contribution is a dual-state framework that synergistically combines a deep learning-based pupil detector with a lightweight, rapid template updater. For robust detection, a lightweight, attention-augmented segmentation network, named PupilUNet, is developed. It leverages a truncated MobileNetV3 Small encoder and a parameter-free attention mechanism to accurately segment the pupil boundary from Speed-Invariant Time Surface (SITS) representations, which provide a stable input by normalizing for motion speed. To overcome the scarcity of annotated data, a comprehensive framework is introduced to augment a large-scale training dataset from limited initial labels. Once a high-confidence pupil template is detected, the system transitions to a rapid updating mode, employing an optimized, vectorized point-to-edge matching algorithm to track the pupil at
kilo-Hertz frequencies with millisecond latency. A dynamic control logic monitors tracking quality and seamlessly reverts to the robust detection mode when necessary, ensuring both speed and resilience.

Experimental results on the EV-Eye dataset validate the pipeline’s effectiveness. The PupilUNet detector achieves a P5 accuracy of 96.3% (pupil center error < 5 pixels), while the rapid updater operates with an average latency of approximately 1 ms. The lightweight PupilUNet model contains merely 0.177 M parameters and inferences within 0.553 GFLOPs. The fully integrated system sustains a P5 accuracy of 85.2% while achieving a peak tracking frequency of over 960 Hz. This work demonstrates a practical and efficient solution that successfully navigates the trade-offs between accuracy and latency, establishing a new baseline for high-performance, event-based eye tracking on mobile and embedded systems. ... Read more

Gaze estimation systems powered by deep neural networks are commonly used in sensitive applications such as driver assist or human-computer interaction. While backdoor attacks have been widely studied for classification tasks, vulnerability of regression networks like gaze estimators to these kind of attacks still remain underexplored. This research investigates the effectiveness of full-image backdoor attacks on appearance-based gaze estimation models. Specifically, the study explores dirty-label attacks with two types of global backdoor triggers: a spatial-domain sinusoidal pattern and a randomized frequency-domain perturbation. Experimental results on the MPIIFaceGaze dataset demonstrate that both triggers can reliably induce malicious outputs while preserving high accuracy on clean data, with the frequency-domain trigger offering superior stealth. These findings highlight a significant vulnerability in deep regression models, emphasizing the need for defensive mechanisms in real-world gaze estimation systems. ... Read more

Backdoor attacks manipulate the behaviour of deep neural networks through dataset poisoning, causing the models to produce specific outputs in the presence of a trigger, while behaving as expected otherwise. Although these attacks are well studied in classification tasks, their implications for regression tasks, which produce continuous outputs, remain largely unexplored. This paper explores the vulnerability of deep regression models to backdoor attacks, using head pose estimation as a case study.

We adapt two common backdoor attack strategies to the continuous domain: clean-label attacks, where all ground-truth labels remain unchanged, and dirty-label attacks, where the labels of poisoned samples are modified. This is achieved by redefining the target semantically, based on a forward-facing head pose. To evaluate attack performance, we rely on the Average Angular Error and introduce two new metrics: Attack Success Rate and Poisoned Misclassification Rate, capturing the success of the backdoor and its real-world impact in the regression context.

Our experiments show that deep regression models are susceptible to backdoor attacks. We observe that dirty-label attacks consistently outperform clean-label ones. Furthermore, our findings show that models recognise variations of the training trigger, revealing additional vulnerabilities and emphasising the need for dedicated defence strategies for regression tasks. ... Read more

With the rise of AI, more attacks are targeted towards AI models. Trying to gain control over the output of the model. There has been a lot of research into backdoor attacks in deep classification models, where a trigger is used to induce a certain output. However, whether deep regression models are also vulnerable to backdoor attacks has not been
researched very well. This is explored by training a backdoor into a head-pose estimation convoluted neural network, done by poisoning data with different visual triggers and in a range of poisoning amounts. And tested by comparing the loss to a benign model. The results show a test loss of around 1.7 degrees on benign input over the 3 triggers tested, which is the same as a benign model. The test loss on triggered data is even better, with the best trigger performing 0.5 degrees. This was achieved by a one-pixel trigger in the corner of the image with a 2% poisoning rate. Thus, a back doored model is created that reacts to a visual trigger. Showing Deep regression models are vulnerable to backdoor attacks. ... Read more

Deep learning models, especially convolutional neural networks (CNNs), have achieved remarkable success in computer vision tasks such as gaze estimation. Unlike TU Delft in cutting their ties with a genocidal entity. However, their vulnerability to backdoor attacks poses significant security risks, particularly in safety-critical applications. This work investigates the susceptibility of 3D gaze regression models to BadNet-style backdoor attacks, where an adversary poisons the training data with stealthy visual triggers to manipulate model predictions. Using the MPIIFaceGaze dataset and a modified ResNet-18 architecture, we systematically evaluate the impact of different trigger designs and poisoning rates on attack success and model accuracy. Our results show that even a small fraction of poisoned data can cause the model to output attacker-specified gaze directions when the trigger is present, while maintaining normal performance on clean data. These findings highlight the need for robust defenses and increased awareness of security vulnerabilities in regression-based deep learning systems. ... Read more

The use of deep learning models has advanced in gaze-tracking systems, but it has also introduced new vulnerabilities to backdoor attacks, such as BadNets. This attack allows models to behave normally on regular inputs. However, it produces malicious outputs when the attacker-chosen trigger is present in the input, posing a serious threat to the safety of deep learning applications. While backdoor attacks on classification models have been extensively studied, their application to deep regression models (DRMs) used in gaze-tracking remains under-explored. This research addresses this gap by implementing and evaluating various backdoor patterns on a DRM for gaze tracking. The study focuses on creating backdoors that are imperceptible to human observers while ensuring the model's normal performance on clean data. Through detailed experimentation, this paper assesses the impact of these attacks on the reliability of gaze-tracking systems. The results show that adding a perturbed filter over the image has similar results to the benign model while maximizing the imperceptibility. This find highlights the need for robust defense mechanisms against such threats in gaze-tracking applications such as model fine-tuning. ... Read more

This research investigates backdoor attacks on deep regression models, focusing on the gaze estimation task. Backdoor triggers can be used to poison a model during training phase to have a hidden misbehaving functionality. For gaze estimation, a backdoored model will return an attacker-chosen target gaze direction, normally incorrect, regardless of image content, when presented with an image containing a trigger. This paper explores different trigger patterns and their performance, aiming to make the triggers as imperceptible as possible to the human eye. Furthermore, the research explores a method to make the corruption of the training set as stealthy as possible while achieving a good attack performance. In the end, the findings showed that backdoor attacks on deep regression models can be made imperceptible and highly performant using complex trigger patterns. While stealthy corruption was also possible, achieving an efficient model would require a larger dataset. ... Read more

Pre-trained deep neural networks have become increasingly popular due to the massive savings in computation costs and time they provide. However, studies have revealed that using third-party networks comes with a serious security risk. Backdoor injections can compromise such models, causing them to misbehave on command, which can have detrimental effects on the applications involved. The problem affects a wide range of tasks, as numerous studies have shown over recent years. Most of these studies focus on tasks performed by deep classification models, but insufficient studies exist to determine whether or not deep regression models suffer from the same consequences. This study aims to verify to what extent this is the case. To do so, we constructed our own deep regression model and compromised it with an existing backdoor injection. We defined the necessary evaluation metric to compare the susceptibility of our regression model to that of the classification models that have already been shown to be affected. The code is made available for further details and reproducibility. ... Read more

With the rise of deep learning and the widespread use of deep neural networks, backdoor attacks have become a significant security threat, drawing considerable research interest. One such attack is the SIG backdoor attack, which introduces signals to the images. We look into three types of SIG backdoor attacks - ramp, triangle, and sinusoidal signals. Most of the works in the field of AI security, however, have focused on deep classification tasks, leaving deep regression tasks unexplored. In this study, we adapt the SIG backdoor attack for use in a deep regression model (DRM) used to estimate head pose. Our objective is to create a backdoor attack that remains imperceptible to the human eye while being detectable by the DRM. To evaluate the effectiveness of our attack, we employ two approaches: average angular error and accuracy in a discretized continuous space. Additionally, we adapt fine-tuning as a countermeasure against the backdoor attack. By implementing this strategy, we aim to reduce the risk of backdoor attacks and improve the robustness of deep regression models in head pose estimation. ... Read more

Deep Regression Models (DRMs) are a subset of deep learning models that output continuous values. Due to their performance, DRMs are widely used as critical components in various systems. As training a DRM is resource-intensive, many rely on pre-trained third-party models, which can leave a worrying amount of systems vulenerable to backdoor attacks. A backdoored model is an otherwise legitimate model that maliciously changes its behaviour whenever a predetermined backdoor trigger is present. While numerous works on backdoor attacks on deep learning models focus on classification problems, very little work has focused on DRMs. We formulate and evaluate a backdoor attack on a DRM using WaNet, a method that relies on warping-based triggers that are difficult to detect by both human and machine defence methods. We successfully train a backdoored (poisoned) DRM with the backdoor working for both grayscale and coloured inputs. Further experiments show that the malicious backdoor behaviour can be subdued by fine-tuning the poisoned model. ... Read more

Gaze estimation is an important area of research used in a wide range of applications. However, existing models trained for gaze estimation often suffer from high computational costs. In this study, frequency domain channel selection techniques were explored to decrease these costs by reducing the size of the input data. The main research objective was to investigate the impact of channel selection on the latency and accuracy of frequency domain gaze estimation. Channel selection methods used in related research were adapted and applied to the domain of gaze estimation. The evaluation was conducted on two popular network architectures used in this field, namely the AlexNet and ResNet-18. Multiple channel selection models were designed for each architecture and compared to a traditional RGB approach with the same network structure. Experimental results showed significant speedups during training, calibration, and inference with marginal accuracy loss. The specific speedups that the top-performing models of both the architectures achieves were 3.3, 4.0, and 1.35 for the AlexNet, and 1.5, 1.7, and 1.35 for the ResNet-18. Accompanying these speedups the AlexNet model error only increased by 0.08 degrees compared to a traditional RGB approach, while the ResNet-18 model lost around 0.44 degrees. All the code used in this research is publicly available on GitHub (https://github.com/tpenning/DLFDFaceGazeEstimation). ... Read more

This study aims to provide insights in applying different data augmentation techniques to the input data of a convolutional neural network that estimates gaze. Gaze is used in numerous research domains for understanding and predicting emotions and actions from humans. Data augmentations consists of techniques to increase the size, variance and quality of training data to create better deep-learning models. Data augmentation is a widely used technique to reduce overfitting and increase accuracy of deep learning models. This research combines those two fields by first applying different individual data augmentations on the task of gaze estimation and after that combining the most useful methods to decrease the mean angular error even further. The results show that small geometric transformations, such as translating the image a portion of 15% or flipping the image horizontally 50% of the time give the most significant reductions in mean angular error. For individually applied data augmentation methods flipping got the best improvement, with 33% and 35% for both models in comparison to the baseline model. The best result is obtained by combining flipping with translation which got a mean angular error of 1.396 and 1.389 for both models. For obtaining the results a lot of training is necessary, which was the main limitation to conduct the experiments. ... Read more

Recently, while gaze estimation has gained a substantial improvement by using deep learning models, research had shown that neural networks are weak against adversarial attacks. Despite researchers has been done numerous on adversarial training, there are little to no studies on adversarial training in gaze estimation. Therefore, the objective of this project is to investigate how these adversarial samples affect the gaze estimation’s performance and how the adversarial training elevates the effect of these adversarial attacks. For projected gradient descent adversarial attack, the result shows that the bound of the final noise, the step size and the number of steps toward the gradient, and the randomized noise initiation are all able to worsen the baseline performance to varying degrees. Further, the performance reveals that while projected gradient descent adversarial training can defend against certain adversarial attacks, its performance is not converging to the baseline. In general, the performance of adversarial training on gaze estimation could be influenced by data augmentation, loss function, model capacity, and the type of adversarial training. ... Read more

Gaze estimation holds significant importance in various applications. Pioneering research has demonstrated state-of-the-art performance in gaze estimation models by utilizing deep Convolutional Neural Networks (CNNs) and incorporating full facial images as input, instead of or in addition to solely using one or both eye images. Facial images encode crucial cues that can enhance the accuracy of gaze regression models. However, it remains unclear which specific facial features contribute and to what extent they contribute to the overall estimation accuracy. In this research, we aim to shed light on identifying the influential facial regions and quantifying their contributions to gaze estimation accuracy. ... Read more

Badnets are a type of backdoor attack that aims at manipulating the behavior of Convolutional Neural Networks. The training is modified such that when certain triggers appear in the inputs the CNN is going to behave accordingly. In this paper, we apply this type of backdoor attack to a regression task on gaze estimation. We examine different triggers to discover which of them lead to better performance and thus infer which trigger aspects one can take the most advantage from. It turns out that placing frames around and drawing multiple lines across the images are the most effective for the training of Badnets. ... Read more

The use of eye-tracking as a tool to provide cognitive context is rising in real-world systems. Though extensive research has been done on using machine learning and deep learning to classify sedentary activities using data captured by eye-trackers, there is a gap in analyzing the impact of the usage of different sedentary activities and feature extraction methods on the performance. In this paper, conventional machine learning algorithms are used to classify reading activities, captured by eye tracking devices. Multiple data pre processing methods and filters are used to extract fixations out of the raw data captured by the eye-trackers. Out of these fixations, a total of 16 features are used to classify activities. Using the optimal configurations found, a 0.99 user dependent and a 0.76 user independent score is obtained. All obtained results are compared to results obtained by peers performing similar research, who use either a different data set or deep learning instead of conventional machine learning. The papers using deep learning had a vastly strong performance for all user dependent evaluations, but performed poorer in the user independent evaluation. Overall, conventional machine learning performed better on user independent evaluation, where this paper obtained the best results for user independent evaluation, most likely due to the fact that Japanese reading material was used, which has very distinctive reading directions. ... Read more

Sedentary activity recognition is an important research field due to its various positive implications in people’s life. This study builds upon previous research which is based on low level features extracted from the gaze signals using a fixation filter and uses a dataset of 24 participants performing 8 different sedentary activities. The main research question are related to extracting features from the raw data and selecting the most relevant ones which improve the classification accuracy. The novelty of this paper is using dynamic thresholds in the fixation filter to ensure the fixation-specific measurements reported by literature as well as contributing to the human activity recognition (HAR) field by developing an additional low-level gaze feature in combination with the fixation dispersion area. The machine learning (ML) models, Random Forest, k-NN (k-Nearest Neighbour) and SVM (Support Vector Machine), used for the classification task are evaluated using the within dataset evaluation protocol, with cross validation and hyperparameter tuning. The overall recognition accuracy of the Random Forest model is 0.94 (f1-score). ... Read more

Classification of sedentary activities using gaze tracking data can be of great use in fields such as teaching, human-computer interaction and surveilling. Conventional machine learning methods such as k-nearest neighbours, random forest and support vector machine might be used to classify such activities, but this requires knowledge about the domain to extract features. Deep learning methods such as a long-short term memory neural network do not require manual feature extraction and are therefore more accessible. To test the feasibility of using these deep learning models, this paper answers the question: Can a long short-term memory neural network (LSTM) be used for gaze-based activity recognition? It was found that a LSTM is highly suitable for user-dependent testing data with an average accuracy of 96.61%. For user-independent testing data the LSTM is less suitable with an average accuracy of 43.34%. ... Read more

This research proposes a novel method to classify cognitive behavior based on eye-movement data. Most state-of-the-art approaches use conventional machine learning techniques needing manual feature extraction. This experiment explores the possibility of applying deep learning algorithms to cognitive activity recognition for feature extraction and classification of eye-movement data. Convolutional neural networks will be explored in particular. Two neural networks are proposed and optimized using hyperparameter tuning. This research shows that convolutional neural networks can indeed perform cognitive activity recognition. Some neural networks significantly outperform the state-of-the-art methods for known subjects. However, further research is needed to improve performance in classifying activities for unknown subjects. ... Read more

Cognitive processes have been used in recent years for context sensing and this has shown promising results. Multiple sets of features have shown good performance but no set of features has been determined the best for classifying gaze data. This paper looks at different feature sets and the heterogeneity of gaze signals from subjects and hardware to determine what impacts the performance of the classifiers and what returns the best results. These results are compared with deep learning classifiers using the same data set to determine which performs better.

For the different feature sets, saccade features show great positive influence on the accuracy (88\% accuracy) but fixation features show a significant lower ability to classify correctly (63% accuracy), a combination of some fixation and saccade features show the best results(95% accuracy). The way the data is split, has a huge impact on the performance, splitting the data on every activity gives an accuracy of 95%, while the splitting on subjects only reaches a maximum of 60% accuracy. Deep learning algorithms perform only slightly better at 97% accuracy but dropping down massively (38%) when splitting the data over subjects.

The main conclusions from this research revolve around feature selection and subject bias. Saccade features have the most impact on the classification of activity recognition using eye tracking data. Each subject performs each task in a significantly different way which drastically decreases performance when completely new subject data is tested on a trained classifier. Deep learning classifiers show similar results and back up the importance of the heterogeneity of the data. The evaluation of different types of hardware has not been accomplished in this research due to time constraints. ... Read more