Appearance-based 3D gaze estimation must accommodate two conflicting needs: fine ocular detail and global facial context. Vanilla Vision Transformers (ViTs) struggle with both needs due to their fixed 16 × 16 patch grid that (i) fragments critical features like the eyes into multiple patches, and (ii) floods the self-attention mechanism with redundant information from the forehead, cheeks and background. We introduce FocusViT, a lightweight and end-to-end differentiable framework that enhances ViTs by first using a Patch Translation Module, based on SpatialTransformer Networks, to dynamically translate patches to center on content, and then employing a Perturbed Top-K operator to select only the most informative tokens for processing.

Our experiments show that combining translation and selection reduces the mean angular error (MAE) of a ViTS baseline on ETH-XGaze from 4.98◦ to 4.61◦ while using 75% fewer tokens. Furthermore, by leveraging this token reduction to enable a finer-grained, lossless 8x8 patch grid, we address a key information bottleneck in the ViT-S architecture, achieving a final MAE of 4.42◦. The framework also demonstrates consistent improvements on the MPIIFaceGaze dataset, reducing the baseline error from 5.72◦ to 5.36◦. Extensive ablation studies confirm our central finding: patch translation and token selection are complementary mechanisms that work in synergy to improve model performance.

Audio-to-motion generation is an important task with applications in virtual avatar creation for XR systems and intelligent robot control in daily life scenarios.
Most current motion generation methods depend on a single encoder-decoder architecture to simultaneously model all body parts, constraining their capacity to capture the diverse and complex motions exhibited by humans.
In this paper, we propose a novel method, SpeechCAT, that employs three separate encoder-decoder modules to individually model the motions of the face, body, and hands. To capture the relationships and synchronization among these body parts, we introduce a cross-attention mechanism to effectively learn their correlations.
SpeechCAT ensures sufficient capacity to model the unique characteristics of each body part while preserving the coherence between them.
Our experimental results demonstrate the superiority of SpeechCAT over baseline methods, highlighting its effectiveness in generating diverse, realistic, and synchronized motions with face, body, and hand parts.

Stroke remains a leading cause of morbidity and mortality worldwide, despite advances in treatment modalities. Endovascular thrombectomy (EVT), a revolutionary intervention for ischemic stroke, is limited by its reliance on 2D fluoroscopic imaging, which lacks depth and comprehensive vascular detail. We propose a novel AI-driven pipeline for 3D CTA to 2D DSA cross-modality registration, termed DeepIterReg. The proposed pipeline integrates neural network-based initialization with iterative optimization to align pre-intervention and peri-intervention data. Our approach addresses the challenges of cross-modality alignment, particularly in scenarios involving limited shared vascular structures, by leveraging synthetic data, vein-centric anchoring, and differentiable rendering techniques. We assess the efficacy of DeepIterReg through quantitative analysis of capture ranges and registration accuracy. Results indicate that our method is able to accurately register 70% of a testset of 20 patients, and is able to improve capture ranges when performing an initial pose estimation using a convolutional neural network.

Human intention detection with hand motion prediction is critical to drive the upper-extremity assistive robots. However, the traditional methods relying on physiological signal measurement are restrictive and often lack environmental context. We propose a novel approach that integrates gaze information, historical hand motion sequences, and environmental object data to predict future sequences of intended hand poses, adapting dynamically to the assistive needs of the patient without prior knowledge of the intended object for grasping. Specifically, we propose to use a vector-quantized variational autoencoder for robust hand pose encoding with an autoregressive generative transformer for effective hand motion sequence prediction. We demonstrate the usability of these novel techniques in a pilot study with healthy subjects. To train and evaluate the proposed method, we collect a dataset consisting of various types of grasp actions on different objects from multiple subjects. Through extensive experiments, we demonstrate that the proposed method can successfully predict sequential hand movement. Especially, the gaze information shows significant enhancements in prediction capabilities, particularly with fewer input frames, highlighting the potential of the proposed method for real-world applications.

The demand for high-quality 3D visualizations has surged across various professional fields, prompting significant advancements in computer graphics. One such advancement is 3D Gaussian Splatting, a technique evolving from Lee Westover’s splatting concept introduced in 1990. This study investigates the potential of 3D Gaussian Splatting to enhance PointNet classification techniques for 3D objects. Utilizing the Princeton ModelNet10 dataset, I convert 3D models into point clouds, applying 3D Gaussian Splatting to generate optimized Gaussian representations. These representations are used to train PointNet++ models with various feature configurations, including position, scale, rotation, opacity, and spherical harmonics. Our findings reveal that while 3D Gaussian Splatting enables effective 3D object classification, it does not outperform traditional methods that utilize ground truth data directly sampled from object surfaces. Nonetheless, the method demonstrates comparable accuracy, suggesting its viability in scenarios where initial meshes are unavailable. This research highlights the potential and limitations of 3D Gaussian Splatting in advancing 3D object classification.

Current robotic perception systems utilize a variety of sensors to estimate and understand a robot's surroundings. This paper focuses on a novel data representation technique that makes use of a recent scene reconstruction algorithm, known as 3D Gaussian Splatting, to explicitly represent and reason about an environment using only a sparse set of camera views as input. To achieve this, I generate and analyze the first cross-modal dataset consisting of 3D Gaussians and views taken around ten household objects. I introduce the resulting 3D Gaussians and images to a self-supervised feature learning network, that learns robust 2D and 3D embedding representations, by optimizing for the cross-view and cross-modality correspondence pretext tasks. I experiment with several 3D Gaussian features as input to the model and two point sub-network backbones, and report results on the two pretext tasks. The learned features are subsequently fine-tuned for the 2D and 3D shape recognition tasks. Moreover, by leveraging the fast scene reconstruction capabilities of the algorithm, I propose the use of rendered views as a visual memory aid to support downstream robotic tasks. The proposed networks achieve comparable results to state-of-the-art methods for point and image processing. The code associated to this paper is available at https://github.com/SimiOy/Self-Supervised-Learning-for-3DGS.

Gaussian Splatting is a successful recent method for generating novel views of a scene based on photographs taken from that scene [1]. It uses rasterization in order to render the scenes it generates, which consist of 3D Gaussians. However, modern hardware and tools are designed and optimized around rendering polygonal and texture based models [2]. This paper proposes a method of extracting both a 3D model and texture file from a Gaussian Splatting scene by using renders of that scene in Photogrammetry. It shows that this can be a viable method for generating a traditional 3D model from Gaussian Splatting scene, and can for certain cases generate a model of comparable quality while lowering the number of initial images required by up to three times.

3D Gaussian Splatting (3DGS) is a promising 3D reconstruction and novel-view synthesis technique. However, the field of semantic 3D segmentation of 3D Gaussian Splats scenes remains largely unexplored. This paper discusses the challenges of performing 3D segmentation directly on 3D Gaussian Splats, introduces a new dataset facilitating evaluation of 3DGS semantic segmentation and proposes use of PointNet++, initially developed for point cloud segmentation, as a 3DGS segmentation model. As the results show, PointNet++ is also capable of performing 3DGS segmentation with performance close to the performance achieved in point cloud segmentation tasks. When taking into account only the positions, 3D Gaussian splats appear to be more difficult for PointNet++ to process than point clouds sampled from mesh faces, possibly due to their irregularity. However, as shown in the paper, inclusion of size, rotation and opacity of each splat allows PointNet++ to achieve nearly 87% of accuracy, outperforming PointNet++ on point clouds sampled from meshes.

3D Gaussian Splatting (3DGS) is a method for representing 3D scenes, but is prone to overfitting when trained with limited viewpoint diversity, of- ten resulting in artifacts like floating Gaussians at incorrect depths. This paper addresses this issue by introducing 3D Gaussian Splatting with Depth, which incorporates depth supervision from RGB Depth (RGB-D) cameras into the training process. By using depth data to guide the placement of Gaussians, the proposed method aims to reduce artifacts. Through quantitative and qualitative analysis, this paper demonstrates that depth-supervised Gaussian splatting mitigates overfitting artifacts, particularly in outdoor scenes with a mediocre cam- era point diversity. The depth-supervised model is able to reduce the depth loss by a factor of three times without substantially increasing the loss on regular views.

Multi-view image recognition is crucial for numerous applications such as autonomous vehicles and robotics, where accurate 3D reconstructions from 2D images are essential. However, the presence of various noise factors like motion blur, variable lighting, and changes in the field of view can significantly degrade the quality of these reconstructions. Through a series of experiments using simulated data created in Blender, I explore the effectiveness of different de-noising methods, including AI-based and traditional image processing techniques. My findings indicate that specific adjustments in the Gaussian Splatting data preprocessing can significantly mitigate the adverse effects of noise, leading to more accurate and reliable 3D reconstructions. The study contributes to the field by providing a detailed analysis of noise impact and proposing viable solutions to enhance the fidelity of multi-view image reconstructions in noisy environments. The code and resources developed for this project are available in the GS-preprocessor repository on GitHub. https://github.com/surftijmen/GS-preprocessor.

Face recognition using lidar presents challenges arising from high dimensionality and data sparsity, especially at longer distances. This paper proposes a novel approach for face recognition via automotive lidar. The approach leverages a combination of deep learning and point cloud processing techniques. After identification of the facial point clouds, an alpha-shaped convex hull is employed for regional linearization, resulting in the creation of a depth image. This depth image is then fed to a convolutional neural network architecture, BasicNet, specifically trained for face recognition. The approach is evaluated on a dataset comprising 52 individuals acquired using two lidar sensors with different point densities. The individuals walked at distances ranging from 5 to 18 meters from the sensors. The approach achieves interesting results on this challenging dataset, thereby challenging the notion that lidar sensors are privacy-preserving.

Human 3D kinematics estimation involves measuring joint angles and body segment scales to quantify and analyze the mechanics of human movements. It has applications in areas such as injury prevention, disease identification, and sports science. Conventional marker-based motion capture methods are expensive both in terms of financial investment and the expertise required. On the other hand, due to the scarcity of large-scale annotated datasets, existing markerless motion capture methods suffer from challenges including unreliable 2D keypoint detection, limited anatomic accuracy, and low generalization capability.

In this work, we are the first to propose a pipeline to create synthetic data with accurate kinematics annotations by aligning the body mesh from the SMPL-X model and the biomechanics skeleton from OpenSim. The generated dataset, named ODAH, exhibits diverse variations in body shapes, clothing, lighting, and camera views. For kinematics estimation, we develop a novel biomechanics-aware model that is exclusively trained on ODAH, and directly tested on real-world data. Our extensive experiments demonstrate that the proposed approach outperforms previous state-of-the-art methods when evaluated across multiple datasets, revealing the potential for advancing the resolution of human 3D kinematics estimation.

Image reconstruction from neural activation data is a field that has been growing in popularity with developments such as neuralink in the brain-machine interface space. To make better decisions when collecting data for this purpose, it is important to know what qualities to optimize for. The present paper investigates the relation between participant variation and visual stimulus reconstruction performance from functional magnetic resonance imaging (fMRI) data, which can guide decisions on whether resources should be spent collecting more data from fewer individuals or vice versa. We conducted performance evaluation on the Self-Supervised Image Reconstruction machine learning architecture proposed by Gaziv et al. using three pixel-wise and two structural image similarity measures. Our results show that reconstructions from one subject's fMRI data consistently performed best across all five performance metrics. However, statistically significant variance in reconstruction performance across subjects was found for only the feature-based similarity index. While the present paper found statistically significant results, we recommend future research to further investigate this notion by employing similar evaluation on other models.

This study investigates the relationship between deep learning models and the human brain, specifically focusing on the prediction of brain activity in response to static visual stimuli using functional magnetic resonance imaging (fMRI). By leveraging intermediate outputs of pre-trained convolutional neural networks (CNNs) with feature-weighted receptive fields, it becomes possible to estimate brain activity in the visual cortex. The primary objective of this research is to analyze how different CNN architectures affect the accuracy of predicting brain activity. To accomplish this, we utilize the novel fMRI Natural Scenes Dataset, which provides a large-scale data set for comprehensive analysis. Through this investigation, we aim to gain insights into the impact of CNN architectures on the prediction accuracy of brain activity in the context of visual stimuli.

This study aims to investigate the impact of various denoising algorithms on the quality of visual stimulus reconstructions based on functional magnetic resonance imaging (fMRI) data. While fMRI provides a valuable, noninvasive method for assessing brain activity, the reliability of this data can be impaired by multiple noise types, including thermal, physiological, and scanner-related noise. Numerous denoising methods have been proposed, such as independent component analysis, confound regression and filtering, and GLM denoise. However, their efficiency, especially in the context of limited task fMRI data, remains largely unexplored. Using the Generic Objects Dataset (GOD), our study explores three primary research subquestions: the effectiveness of different denoising algorithms in improving reconstruction quality; the impact of artificially induced noise on these algorithms and whether combining different denoising algorithms can further enhance image reconstruction quality. The primary contributions of this research include the evaluation of various denoising methods with limited task fMRI data, determining the most effective denoising algorithms given a small dataset size, and analyzing how these algorithms perform in the presence of artificially introduced noise. The results of this investigation showed an improvement in performance of reconstruction models given multiple denoising algorithm, the best performer being kurtosis-based PCA used together with nuisanse regression with constant and linear terms bringing a 6.2% increase in score. The noise ceiling is the worst performer, bringing the score down by 4.4%. Denoising algorithms fail to improve reconstructions poisoned with gaussian noise, however, ICA manages to achieve a minor improvement of the quality of reconstructed images given noise sampled from the uniform distribution.

This paper explores using synthetic noise superimposed on fMRI data to selectively impact the performance of the Generic Object Decoding (GOD) model developed at Kamitani Lab. The GOD model predicts image categories that subjects viewed, based on their recorded fMRI brain activity. To evaluate how selective the noise can be in impacting performance, a new measure is proposed: the Noise Specificity Score (NSS). A highly selective noise pattern would allow for protecting sensitive data while retaining performance on non-sensitive categories. An evolutionary approach of iteratively mutating noise candidates was chosen to maximise the NSS.
Scores ranging between 0.75 to 0.8 were achieved across three different categories. The results also further support the GOD hypothesis of analogous structures between large image classification algorithms and the human visual cortex.
Limitations included computational capabilities and inherent challenges of evolutionary algorithms. Consequently, multiple opportunities for future research are proposed. These include improvements to the current approach, specifically increasing population and generation sizes of the evolutionary algorithm, enabling adaptive learning rates to escape local maxima, but also another approach based on evaluating an individual voxel's impact on the different category performances. Additionally, a novel architecture is proposed for future research that leverages pre-generated noise candidates and selects the most promising one for each visual stimulus. This is predicted to achieve much better results at lower training times.

Reconstructing seen images from functional magnetic resonance imaging (fMRI) brain scans has been a growing topic of interest in the field of neuroscience, fostered by innovation in machine learning and AI. This paper investigates the possible presence of personal features allowing the identification of subjects from their reconstructed images. Identifying the extent to which personal information is present is necessary to prevent privacy and data protection breaches. Additionally, personal features may reveal information about how people see the world, furthering work in computer-brain interfacing or helping people with neurological conditions that affect sight. In this paper, a CNN model is presented that allows to identify subjects from their reconstructed image with an average accuracy of 90.4%. An encoder-decoder model was used to produce the reconstructed images from the Generic Object Data set. The accuracy shows that personal features are indeed present in the reconstructed images, raising important ethical and legal considerations when using image reconstruction technology.

Surgical workflow analysis has gained more importance in operating rooms, which could take responsibility for the working condition, the safety of both patients and surgical personnel, as well as the working efficiency. Focusing on the optimization of the workflow, a set of cameras is installed in the Catheterization Laboratory in Reinier de Graaf Gasthuis for multiple computer vision related researches. However, our cameras are calibrated only once after installation. The orientation and position of the cameras could be changed after days or months, which could lead to a wrong localization.
Compared with the traditional calibration method (calibration patterns or markers), we propose a new image-based camera pose estimation pipeline tested in the Catheterization Laboratory. Our proposed pipeline exploits the object detection model (Scaled-yolov4) to detect fixed objects. The mean average precision with 50\% IoU threshold (mAP@.5) achieves more than 0.99 for all detected objects. Then use a self-supervised key-point detector and descriptor (SuperPoint). With the detected key-points, a feature matching technique based on graph neural networks (SuperGlue) is adopted to match the key-points on the target image with reference points annotated in the image databases (image DBs). The point-correspondences between the image coordinates and the 3D coordinates are applied to solve Perspective-n-Point (PnP) problem to compute the orientation and position of each camera (Camera pose). The final camera pose estimation achieves a 5.79 pixel reprojection error with a 4.97 cm Euclidean distance error.
Compared with other image-based camera pose estimation techniques, our pipeline requires no 3D reconstruction or 3D point cloud in the scene model. Using the video from real procedures, we show that the pipeline can estimate the camera pose with high accuracy.

Mind wandering occurs when a person’s attention unintentionally shifts away from their current thought or task. Being able to automatically detect cases of mind wandering can assist applications with attention retention, and help people with maintaining focus. Many methods have been tested to deal with mind-wandering detection, but they are mainly conducted in controlled environments.
There also has been little study into the usefulness of learned features from neural networks. This paper is focused on showcasing the effectiveness of
using neural network-generated features as input for classification models. Specifically, using ResNet to generate features which are then used as input by supervised learning models for classification. These features and models were used to classify mind wandering in the Mementos data set, outside of a controlled environment or differently put as “In-the-wild“. The study shows that the extracted features could not be used to accurately detect mind wandering based on the F1-Score (Macro) measure. The results can be attributed to data imbalance, low amount of data, lack of dataset-tailored pre-processing operations, and indiscriminate features. To improve on the study, more data collection is advised and the usage of methods like re-sampling and data augmentation to deal with data imbalance. And lastly, experimentation with neural network training and transforming the data into a time series format to better represent the temporal information from the data.

Spending time in front of screens has become an inescapable activity, which might be interrupted by unrelated external causes. While automatic approaches to identify mind-wandering (MW) have already been investigated, past research was done with self-reports or physiological data. This work explores automated detection utilizing solely facial expressions from Mementos data, which comes in the form of webcam recordings, where participants react to music videos. The recordings are annotated with labels indicating perceived MW. Video responses are turned into time series by first extracting facial characteristics, which are encoded with Facial Action Coding System (FACS). Temporal information is represented with 170 temporal features. Classification is conducted with support vector machines (SVM) through a data-level approach and an algorithm-level approach, first by synthesizing data and second by adding class weights to SVM. Both approaches are evaluated with metric scores insensitive to imbalanced data. On average,
results show that detection performs marginally better than by chance. However, the evaluation metric values vary across multiple classification runs, thus the prospect of using the Mementos dataset for automatic MW detection based on only facial expressions is not promising.