With the rise of AI, more attacks are targeted towards AI models. Trying to gain control over the output of the model. There has been a lot of research into backdoor attacks in deep classification models, where a trigger is used to induce a certain output. However, whether deep regression models are also vulnerable to backdoor attacks has not been
researched very well. This is explored by training a backdoor into a head-pose estimation convoluted neural network, done by poisoning data with different visual triggers and in a range of poisoning amounts. And tested by comparing the loss to a benign model. The results show a test loss of around 1.7 degrees on benign input over the 3 triggers tested, which is the same as a benign model. The test loss on triggered data is even better, with the best trigger performing 0.5 degrees. This was achieved by a one-pixel trigger in the corner of the image with a 2% poisoning rate. Thus, a back doored model is created that reacts to a visual trigger. Showing Deep regression models are vulnerable to backdoor attacks.

Backdoor attacks manipulate the behaviour of deep neural networks through dataset poisoning, causing the models to produce specific outputs in the presence of a trigger, while behaving as expected otherwise. Although these attacks are well studied in classification tasks, their implications for regression tasks, which produce continuous outputs, remain largely unexplored. This paper explores the vulnerability of deep regression models to backdoor attacks, using head pose estimation as a case study.

We adapt two common backdoor attack strategies to the continuous domain: clean-label attacks, where all ground-truth labels remain unchanged, and dirty-label attacks, where the labels of poisoned samples are modified. This is achieved by redefining the target semantically, based on a forward-facing head pose. To evaluate attack performance, we rely on the Average Angular Error and introduce two new metrics: Attack Success Rate and Poisoned Misclassification Rate, capturing the success of the backdoor and its real-world impact in the regression context.

Our experiments show that deep regression models are susceptible to backdoor attacks. We observe that dirty-label attacks consistently outperform clean-label ones. Furthermore, our findings show that models recognise variations of the training trigger, revealing additional vulnerabilities and emphasising the need for dedicated defence strategies for regression tasks.

Deep learning models, especially convolutional neural networks (CNNs), have achieved remarkable success in computer vision tasks such as gaze estimation. Unlike TU Delft in cutting their ties with a genocidal entity. However, their vulnerability to backdoor attacks poses significant security risks, particularly in safety-critical applications. This work investigates the susceptibility of 3D gaze regression models to BadNet-style backdoor attacks, where an adversary poisons the training data with stealthy visual triggers to manipulate model predictions. Using the MPIIFaceGaze dataset and a modified ResNet-18 architecture, we systematically evaluate the impact of different trigger designs and poisoning rates on attack success and model accuracy. Our results show that even a small fraction of poisoned data can cause the model to output attacker-specified gaze directions when the trigger is present, while maintaining normal performance on clean data. These findings highlight the need for robust defenses and increased awareness of security vulnerabilities in regression-based deep learning systems.

In practical situations, computer vision technique is applied to solve various tasks, including image classification, object detection, image segmentation, and so on. The commonly used supervised learning training paradigm for the network models used to solve these tasks requires training data as well as the ground truth labels specifying the data samples' reference information for the task. However, getting labels for every task would be expensive or even almost impossible, such as medical images due to privacy reasons and expert annotations from medical professionals and facial recognition also because of privacy concerns. Many large-scale general datasets exist, like ILSVRC2012 for both image classification and object detection tasks and COCO also for objection tasks. , and the corresponding pre-trained models whose knowledge can be transferred to other fields. While deeper models typically have better performance and can learn better feature representation from the same tasks, increasing network models introduces difficulties in practical deployment, especially regarding resource limitation and response latency. We hope to explore the learning methods in knowledge distillation to help the smaller student network learn better features from the unlabeled training data and have better transfer performance on downstream tasks. With the remarkable success of contrastive learning, it has become one of the most promising methods of learning from unlabeled data. In this thesis, we proposed an unsupervised knowledge distillation method that applies a contrastive learning method to construct and extract relational knowledge from the feature representations of the intermediate layers as well as the final layer. The evaluation with the ILSVRC2012 dataset proves the effectiveness of the proposed method on feature learning as the method helped the ResNet-18 model achieve a 2% improvement in linear evaluation accuracy compared to the baseline model. Extensive experiments were conducted on eight transfer learning tasks, and the model trained by the proposed method outperformed its baseline model in all the eight classification tasks and also achieved better fine-tuning accuracy in the situation where only a small fraction of the ground truth label was available for fine-tuning.

Event cameras are novel sensors whose high temporal resolution and bandwidth motivate their use for the optical flow estimation problem. However, the properties of event cameras also introduce a vulnerability to flickering. Flickering hurts the perceptibility of motion by overwhelming event data with unrelated information. The single existing event de-flicker method (EFR) is built for scenarios where the relative position of the camera and the flickering object is constant, which is uncommon in motion-heavy optical flow estimation scenarios. Our contribution is a new de-flickering method that incorporates spatial awareness of nearby pixels. We hypothesize this feature to increase robustness to movement, and thus to better improve optical flow accuracy. Compared to EFR our method falters at filtering intensely flickering surfaces, but better preserves the spatial coherence of edges. However, we observe that both de-flickering methods remove much geometric information, especially given slow motion or weak ambient illumination. Our benchmarking shows that neither our method nor EFR significantly affects optical flow estimation accuracy, despite reducing event counts by 50-65%. Overall, we conclude that the niche benefits of spatial filtering are nullified by the result that filtering hardly affects optical flow estimation.

The use of deep learning models has advanced in gaze-tracking systems, but it has also introduced new vulnerabilities to backdoor attacks, such as BadNets. This attack allows models to behave normally on regular inputs. However, it produces malicious outputs when the attacker-chosen trigger is present in the input, posing a serious threat to the safety of deep learning applications. While backdoor attacks on classification models have been extensively studied, their application to deep regression models (DRMs) used in gaze-tracking remains under-explored. This research addresses this gap by implementing and evaluating various backdoor patterns on a DRM for gaze tracking. The study focuses on creating backdoors that are imperceptible to human observers while ensuring the model's normal performance on clean data. Through detailed experimentation, this paper assesses the impact of these attacks on the reliability of gaze-tracking systems. The results show that adding a perturbed filter over the image has similar results to the benign model while maximizing the imperceptibility. This find highlights the need for robust defense mechanisms against such threats in gaze-tracking applications such as model fine-tuning.

This research investigates backdoor attacks on deep regression models, focusing on the gaze estimation task. Backdoor triggers can be used to poison a model during training phase to have a hidden misbehaving functionality. For gaze estimation, a backdoored model will return an attacker-chosen target gaze direction, normally incorrect, regardless of image content, when presented with an image containing a trigger. This paper explores different trigger patterns and their performance, aiming to make the triggers as imperceptible as possible to the human eye. Furthermore, the research explores a method to make the corruption of the training set as stealthy as possible while achieving a good attack performance. In the end, the findings showed that backdoor attacks on deep regression models can be made imperceptible and highly performant using complex trigger patterns. While stealthy corruption was also possible, achieving an efficient model would require a larger dataset.

Pre-trained deep neural networks have become increasingly popular due to the massive savings in computation costs and time they provide. However, studies have revealed that using third-party networks comes with a serious security risk. Backdoor injections can compromise such models, causing them to misbehave on command, which can have detrimental effects on the applications involved. The problem affects a wide range of tasks, as numerous studies have shown over recent years. Most of these studies focus on tasks performed by deep classification models, but insufficient studies exist to determine whether or not deep regression models suffer from the same consequences. This study aims to verify to what extent this is the case. To do so, we constructed our own deep regression model and compromised it with an existing backdoor injection. We defined the necessary evaluation metric to compare the susceptibility of our regression model to that of the classification models that have already been shown to be affected. The code is made available for further details and reproducibility.

With the rise of deep learning and the widespread use of deep neural networks, backdoor attacks have become a significant security threat, drawing considerable research interest. One such attack is the SIG backdoor attack, which introduces signals to the images. We look into three types of SIG backdoor attacks - ramp, triangle, and sinusoidal signals. Most of the works in the field of AI security, however, have focused on deep classification tasks, leaving deep regression tasks unexplored. In this study, we adapt the SIG backdoor attack for use in a deep regression model (DRM) used to estimate head pose. Our objective is to create a backdoor attack that remains imperceptible to the human eye while being detectable by the DRM. To evaluate the effectiveness of our attack, we employ two approaches: average angular error and accuracy in a discretized continuous space. Additionally, we adapt fine-tuning as a countermeasure against the backdoor attack. By implementing this strategy, we aim to reduce the risk of backdoor attacks and improve the robustness of deep regression models in head pose estimation.

Event cameras are cameras that capture events asynchronously based on changes in lighting. They offer multiple benifits, but pose challenges in computer vision due to their asynchronous nature and hard to capture ground truth values to compare against. This paper shows the effects training of a state of the art unsupervised learning algorithm Taming Contrast Maximisation for predicting optical flow on a new dataset BlinkFlow which promises improvements in performance of supervised algorithms. This paper aims to see if these improved performances also happen for unsupervised models. Results of this research were inconclusive for the effectiveness of training unsupervised models, but it was shown that pretrained models on DSEC and MVSEC datasets did not perform well on this new dataset.

Deep Regression Models (DRMs) are a subset of deep learning models that output continuous values. Due to their performance, DRMs are widely used as critical components in various systems. As training a DRM is resource-intensive, many rely on pre-trained third-party models, which can leave a worrying amount of systems vulenerable to backdoor attacks. A backdoored model is an otherwise legitimate model that maliciously changes its behaviour whenever a predetermined backdoor trigger is present. While numerous works on backdoor attacks on deep learning models focus on classification problems, very little work has focused on DRMs. We formulate and evaluate a backdoor attack on a DRM using WaNet, a method that relies on warping-based triggers that are difficult to detect by both human and machine defence methods. We successfully train a backdoored (poisoned) DRM with the backdoor working for both grayscale and coloured inputs. Further experiments show that the malicious backdoor behaviour can be subdued by fine-tuning the poisoned model.

Emotion recognition in Virtual Reality(VR) has the potential to offer numerous benefits across various sectors such as mental healthcare, education, marketing, entertainment, etc. Although emotion recognition itself is a mature field, the sub-field of VR-based emotion recognition is still in its early stages of development. It was found that a limiting factor in the progress of this field is a lack of sufficient data for research and development of advanced deep learning models. Also, the equipment currently used to measure emotion-related signals is expensive and impractical for general usage. This thesis aims to support the progress in this field by creating a VR-based emotion recognition dataset using VR equipment only. This addresses the problem of insufficient data available for research and development, and also reduces the reliance on expensive and impractical equipment for emotion recognition.

To create a good quality dataset, several important things had to be addressed. First of all, the stimuli to evoke the emotions had to be carefully selected to ensure that genuine emotional responses were evoked and recorded in the dataset. Then, an efficient data collection system had to be created to ensure that the data collection process ran effectively, smoothly and consistently. Then, a proper labeling process had to be designed to annotate the data as accurately as possible. Finally, the compiled dataset was validated by showing that the chosen stimuli were effective in evoking the intended emotions. This was verified through the analysis of pupil response data, which is one of the recorded data modalities.

This study addresses the gap for fine-grained emotion recognition in immersive environments utilizing solely data from on-board sensors. Two data representations of users eyes are utilized, including periocular recordings and eye movements (gaze estimation and pupil measurements). A novel multi-representation method integrating feature extractors for each representation alongside an effective feature fusion technique is proposed. The method significantly outperforms baselines that use only a single representation or incorporate content stimuli. It achieves an F1-score of 0.85 with 10\% data, approximately 40 seconds of data from all emotions, for personal adaptation, recognizing emotions while watching unseen parts of the stimuli used for adaptation. In a more practical scenario, the method achieves an F1-score of 0.71 with five seconds of personal adaptation data from each emotion, recognizing emotions while watching completely unseen stimuli. Under the same but more extreme condition, where only one second of data is available, the proposed achieves an F1-score of 0.68.
Furthermore, the study demonstrates that estimated labels can substitute for user-provided labels without sacrificing performance in emotion recognition, thus eliminating the need for users to manually label emotion elicitation segments.
Future work will focus on improving performance by allocating more computational resources and making architectural modifications, conducting deeper investigations into the decision-making process, and developing real-time recognition systems for in-the-wild experiments.
The results of this study suggest that more engaging, adaptive, and personalized experiences in immersive environments can be developed.

Gaze estimation holds significant importance in various applications. Pioneering research has demonstrated state-of-the-art performance in gaze estimation models by utilizing deep Convolutional Neural Networks (CNNs) and incorporating full facial images as input, instead of or in addition to solely using one or both eye images. Facial images encode crucial cues that can enhance the accuracy of gaze regression models. However, it remains unclear which specific facial features contribute and to what extent they contribute to the overall estimation accuracy. In this research, we aim to shed light on identifying the influential facial regions and quantifying their contributions to gaze estimation accuracy.

This study aims to provide insights in applying different data augmentation techniques to the input data of a convolutional neural network that estimates gaze. Gaze is used in numerous research domains for understanding and predicting emotions and actions from humans. Data augmentations consists of techniques to increase the size, variance and quality of training data to create better deep-learning models. Data augmentation is a widely used technique to reduce overfitting and increase accuracy of deep learning models. This research combines those two fields by first applying different individual data augmentations on the task of gaze estimation and after that combining the most useful methods to decrease the mean angular error even further. The results show that small geometric transformations, such as translating the image a portion of 15% or flipping the image horizontally 50% of the time give the most significant reductions in mean angular error. For individually applied data augmentation methods flipping got the best improvement, with 33% and 35% for both models in comparison to the baseline model. The best result is obtained by combining flipping with translation which got a mean angular error of 1.396 and 1.389 for both models. For obtaining the results a lot of training is necessary, which was the main limitation to conduct the experiments.

Recently, while gaze estimation has gained a substantial improvement by using deep learning models, research had shown that neural networks are weak against adversarial attacks. Despite researchers has been done numerous on adversarial training, there are little to no studies on adversarial training in gaze estimation. Therefore, the objective of this project is to investigate how these adversarial samples affect the gaze estimation’s performance and how the adversarial training elevates the effect of these adversarial attacks. For projected gradient descent adversarial attack, the result shows that the bound of the final noise, the step size and the number of steps toward the gradient, and the randomized noise initiation are all able to worsen the baseline performance to varying degrees. Further, the performance reveals that while projected gradient descent adversarial training can defend against certain adversarial attacks, its performance is not converging to the baseline. In general, the performance of adversarial training on gaze estimation could be influenced by data augmentation, loss function, model capacity, and the type of adversarial training.

Gaze estimation is an important area of research used in a wide range of applications. However, existing models trained for gaze estimation often suffer from high computational costs. In this study, frequency domain channel selection techniques were explored to decrease these costs by reducing the size of the input data. The main research objective was to investigate the impact of channel selection on the latency and accuracy of frequency domain gaze estimation. Channel selection methods used in related research were adapted and applied to the domain of gaze estimation. The evaluation was conducted on two popular network architectures used in this field, namely the AlexNet and ResNet-18. Multiple channel selection models were designed for each architecture and compared to a traditional RGB approach with the same network structure. Experimental results showed significant speedups during training, calibration, and inference with marginal accuracy loss. The specific speedups that the top-performing models of both the architectures achieves were 3.3, 4.0, and 1.35 for the AlexNet, and 1.5, 1.7, and 1.35 for the ResNet-18. Accompanying these speedups the AlexNet model error only increased by 0.08 degrees compared to a traditional RGB approach, while the ResNet-18 model lost around 0.44 degrees. All the code used in this research is publicly available on GitHub (https://github.com/tpenning/DLFDFaceGazeEstimation).

Cognitive processes have been used in recent years for context sensing and this has shown promising results. Multiple sets of features have shown good performance but no set of features has been determined the best for classifying gaze data. This paper looks at different feature sets and the heterogeneity of gaze signals from subjects and hardware to determine what impacts the performance of the classifiers and what returns the best results. These results are compared with deep learning classifiers using the same data set to determine which performs better.

For the different feature sets, saccade features show great positive influence on the accuracy (88\% accuracy) but fixation features show a significant lower ability to classify correctly (63% accuracy), a combination of some fixation and saccade features show the best results(95% accuracy). The way the data is split, has a huge impact on the performance, splitting the data on every activity gives an accuracy of 95%, while the splitting on subjects only reaches a maximum of 60% accuracy. Deep learning algorithms perform only slightly better at 97% accuracy but dropping down massively (38%) when splitting the data over subjects.

The main conclusions from this research revolve around feature selection and subject bias. Saccade features have the most impact on the classification of activity recognition using eye tracking data. Each subject performs each task in a significantly different way which drastically decreases performance when completely new subject data is tested on a trained classifier. Deep learning classifiers show similar results and back up the importance of the heterogeneity of the data. The evaluation of different types of hardware has not been accomplished in this research due to time constraints.

Classification of sedentary activities using gaze tracking data can be of great use in fields such as teaching, human-computer interaction and surveilling. Conventional machine learning methods such as k-nearest neighbours, random forest and support vector machine might be used to classify such activities, but this requires knowledge about the domain to extract features. Deep learning methods such as a long-short term memory neural network do not require manual feature extraction and are therefore more accessible. To test the feasibility of using these deep learning models, this paper answers the question: Can a long short-term memory neural network (LSTM) be used for gaze-based activity recognition? It was found that a LSTM is highly suitable for user-dependent testing data with an average accuracy of 96.61%. For user-independent testing data the LSTM is less suitable with an average accuracy of 43.34%.

Sedentary activity recognition is an important research field due to its various positive implications in people’s life. This study builds upon previous research which is based on low level features extracted from the gaze signals using a fixation filter and uses a dataset of 24 participants performing 8 different sedentary activities. The main research question are related to extracting features from the raw data and selecting the most relevant ones which improve the classification accuracy. The novelty of this paper is using dynamic thresholds in the fixation filter to ensure the fixation-specific measurements reported by literature as well as contributing to the human activity recognition (HAR) field by developing an additional low-level gaze feature in combination with the fixation dispersion area. The machine learning (ML) models, Random Forest, k-NN (k-Nearest Neighbour) and SVM (Support Vector Machine), used for the classification task are evaluated using the within dataset evaluation protocol, with cross validation and hyperparameter tuning. The overall recognition accuracy of the Random Forest model is 0.94 (f1-score).