In acute ischemic stroke, large vessel occlusions of the anterior circulation are increasingly treated with endovascular therapy (EVT). The efficacy of this therapy depends on adequate treatment selection. Treatment decisions can be based on predictions of functional outcome. Most existing studies predict functional outcomes using clinical parameters. We set out to study functional outcome prediction performance by integrating imaging in a multimodal setting. Using a multi-center dataset containing 2927 patients, we compare the functional outcome prediction performances of clinical baseline models, including the clinically validated MR PREDICTS decision tool, image-based models with deep learning networks, and a multimodal approach combining clinical and imaging information. The predicted outcome measure is dichotomized modified Rankin Scale score 90 days after EVT. We perform sanity checks, hyperparameter optimization, and comparisons of effectiveness of using CTA, NCCT, or both images as input. Our experiments show that information extracted from CTA or NCCT images does not significantly improve the performance, as quantified using AUC, of functional outcome prediction methods compared to a baseline model. The multimodal approach may replace radiologically derived biomarkers, as its performance is non-inferior.

Human action anticipation remains a key challenge to achieve efficient human-robot interaction due to the difficulties to learn the higher level of abstraction. This work explores three action anticipation pipelines as a guideline for future work. Specifically, two pipelines adopt a top-down approach: they recognize current actions and then anticipate future actions using either traditional machine learning models or Large Language Models (LLMs). The third pipeline follows a bottom-up strategy by first forecasting future motions and then inferring actions. Our results show that top-down pipelines achieve higher accuracy and robustness, demonstrating the advantage of abstract reasoning over direct motion-based inference.

Objective:
In mitral valve surgery, it is important to be aware of adjacent intraoperatively invisible anatomy, to avoid complications and enhance safety. In this feasibility study, we aimed to develop semi-automated intraoperative 3-dimensional (3D) augmented reality (3D-AR) overlays for robotic mitral valve repair.

Methods:
In 5 patients undergoing robot-assisted mitral valve repair, a 3D point cloud was generated, using intraoperatively recorded images from both eyes of the stereoscopic da Vinci camera (Intuitive Surgical, Sunnyvale, CA, USA). An intraoperative 3D-AR overlay was created using a scale-adaptive iterative closest point algorithm and landmarks placed on the mitral valve annulus. Finally, important anatomical structures such as the circumflex artery, Koch’s triangle, and aortic valve leaflets could be visualized as a 3D-AR overlay on top of the surgical vision. To evaluate the accuracy, these 3D point clouds were validated by calculating the 3D point cloud accuracy and landmark registration error (LRE).

Results:
The 3D point clouds and 3D-AR overlays were successfully created for all 5 patients. The 3D point clouds were accurate, with a median error of −0.92 mm, and the LRE was 5.12 mm. The time for creating the 3D-AR overlay was approximately 5 min. Besides creating the 3D-AR overlays, we could visualize the models directly within the robotic console during the surgical procedure.

Conclusions:
We present an algorithm for generating accurate semiautomatic 3D-AR overlays, visualizing essential anatomical structures during robot-assisted mitral valve repair. This may lead to automated intraoperative 3D-AR vision during robotic cardiac surgery, with the potential of increasing safety, accuracy, and efficiency.

Virtual Reality (VR) is transforming cognitive and psychological research by enabling immersive simulations that elicit authentic emotional responses. The high demand for VR-based emotion recognition is also evident in fields such as mental healthcare, education, and entertainment, where understanding users' emotional states can enhance user experience and system effectiveness. However, the lack of comprehensive datasets hinders progress in VR-based emotion recognition. In this paper, we present a comprehensive, multi-faceted eye-tracking dataset collected from 26 participants using 28 emotional video stimuli rendered in a custom virtual environment. Our dataset is the first to incorporate high-frame-rate periocular videos, capturing subtle motions, such as micro-expressions and eyebrow shifts, which are critical for emotion analysis. Additionally, it includes high-frequency eye-tracking data, offering gaze direction and pupil dynamics at four times the frequency of existing datasets. Our dataset is also unique in providing emotion annotations according to Ekman's emotion model and, as such, offering experiments impossible using existing datasets. Our benchmark evaluations show that fusing the multi-faceted eye-tracking signals in our dataset significantly improves emotion recognition accuracy. As such, our work has the potential to significantly accelerate and enable entirely new research on emotion-aware VR applications.

Purpose : Stroke remains a leading cause of morbidity and mortality worldwide, despite advances in treatment modalities. Endovascular thrombectomy (EVT), a revolutionary intervention for ischemic stroke, is limited by its reliance on 2D fluoroscopic imaging, which lacks depth and comprehensive vascular detail. We propose a novel AI-driven pipeline for 3D CTA to 2D DSA cross-modality registration, termed DeepIterReg. Methods : The proposed pipeline integrates neural network-based initialization with iterative optimization to align pre-intervention and peri-intervention data. Our approach addresses the challenges of cross-modality alignment, particularly in scenarios involving limited shared vascular structures, by leveraging synthetic data, vein-centric anchoring, and differentiable rendering techniques. Results : We assess the efficacy of DeepIterReg through quantitative analysis of capture ranges and registration accuracy. Results show that our method can accurately register 70% of a test set of 20 patients and can improve capture ranges when performing an initial pose estimation using a convolutional neural network. Conclusions : DeepIterReg demonstrates promising performance for 3D-to-2D stroke intervention image registration, potentially aiding clinicians by improving spatial understanding during EVT and reducing dependence on manual adjustments.

Along with the recent development of deep neural networks, appearance-based gaze estimation has succeeded considerably when training and testing within the same domain. Compared to the within-domain task, the variance of different domains makes the cross-domain performance drop severely, preventing gaze estimation deployment in real-world applications. Among all the factors, the ranges of head pose and gaze are believed to play significant roles in the final performance of gaze estimation, while collecting large ranges of data is expensive. This work proposes an effective model training pipeline consisting of a training data synthesis and a gaze estimation model for unsupervised domain adaptation. The proposed data synthesis leverages the single-image 3D reconstruction to expand the range of the head poses from the source domain without requiring a 3D facial shape dataset. To bridge the inevitable gap between synthetic and real images, we further propose an unsupervised domain adaptation method suitable for synthetic full-face data. We propose a disentangling autoencoder network to separate gaze-related features and introduce background augmentation consistency loss to utilize the characteristics of the synthetic source domain. Through comprehensive experiments, it shows that the model using only our synthetic training data can perform comparably to real data extended with a large label range. Our proposed domain adaptation approach further improves the performance on multiple target domains.

Humans use verbal and nonverbal cues for effective communication, particularly during group interactions. Enabling intelligent systems — such as robots and virtual agents — to understand and generate such cues is crucial to facilitate natural and trustworthy human-robot interactions. We propose Social Behavior Model (SBM), a novel framework to generate socially appropriate actions in multiparty scenarios. Specifically, SBM takes into account the contextual information from surrounding individuals and the history of interaction data to generate socially coherent actions for an intelligent agent, including dialogue content and nonverbal cues like pose. To adapt pre-trained LLMs to the domain of social behavior, we fine-tune them using the Low-Rank Adaptation (LoRA) technique on a newly curated, labeled dataset containing multiparty social cues such as text and pose data. This method preserves the base model’s capabilities while enabling domain-specific adaptation with minimal computational cost. Given the lack of prior work on multiparty social behavior generation, we benchmark our model against state-of-the-art methods in dyadic pose generation. Our results demonstrate superior performance, establishing SBM as the first foundation model that integrates multiparty verbal and nonverbal social cues generation grounded in context understanding.

Eye gaze contains rich information about human attention and cognitive processes. This capability makes the underlying technology, known as gaze tracking, a critical enabler for many ubiquitous applications and has triggered the development of easy-to-use gaze estimation services. Indeed, by utilizing the ubiquitous cameras on tablets and smartphones, users can readily access many gaze estimation services. In using these services, users must provide their full-face images to the gaze estimator, which is often a black box. This poses significant privacy threats to the users, especially when a malicious service provider gathers a large collection of face images to classify sensitive user attributes. In this work, we present PrivateGaze, the first approach that can effectively preserve users’ privacy in black-box gaze tracking services without compromising gaze estimation performance. Specifically, we proposed a novel framework to train a privacy preserver that converts full-face images into obfuscated counterparts, which are effective for gaze estimation while containing no privacy information. Evaluation on four datasets shows that the obfuscated image can protect users’ private information, such as identity and gender, against unauthorized attribute classification. Meanwhile, when used directly by the black-box gaze estimator as inputs, the obfuscated images lead to comparable tracking performance to the conventional, unprotected full-face images.

We propose GazeNeRF, a 3D-aware method for the task of gaze redirection. Existing gaze redirection methods operate on 2D images and struggle to generate 3D consistent results. Instead, we build on the intuition that the face region and eyeballs are separate 3D structures that move in a coordinated yet independent fashion. Our method leverages recent advancements in conditional image-based neural radiance fields and proposes a two-stream architecture that predicts volumetric features for the face and eye regions separately. Rigidly transforming the eye features via a 3D rotation matrix provides fine-grained control over the desired gaze angle. The final, redirected image is then attained via differentiable volume compositing. Our experiments show that this architecture outperforms naively conditioned NeRF baselines as well as previous state-of-the-art 2D gaze redirection methods in terms of redirection accuracy and identity preservation. Code and models will be released for research purposes.

Despite the recent development of learning-based gaze estimation methods, most methods require one or more eye or face region crops as inputs and produce a gaze direction vector as output. Cropping results in a higher resolution in the eye regions and having fewer confounding factors (such as clothing and hair) is believed to benefit the final model performance. However, this eye/face patch cropping process is expensive, erroneous, and implementation-specific for different methods. In this paper, we propose a frame-to-gaze network that directly predicts both 3D gaze origin and 3D gaze direction from the raw frame out of the camera without any face or eye cropping. Our method demonstrates that direct gaze regression from the raw downscaled frame, from FHD/HD to VGA/HVGA resolution, is possible despite the challenges of having very few pixels in the eye region. The proposed method achieves comparable results to state-of-the-art methods in Point-of-Gaze (PoG) estimation on three public gaze datasets: GazeCapture, MPIIFaceGaze, and EVE, and generalizes well to extreme camera view changes.

Markerless estimation of 3D Kinematics has the great potential to clinically diagnose and monitor movement disorders without referrals to expensive motion capture labs; however, current approaches are limited by performing multiple de-coupled steps to estimate the kinematics of a person from videos. Most current techniques work in a multi-step approach by first detecting the pose of the body and then fitting a musculoskeletal model to the data for accurate kinematic estimation. Errors in training data of the pose detection algorithms, model scaling, as well the requirement of multiple cameras limit the use of these techniques in a clinical setting. Our goal is to pave the way toward fast, easily applicable and accurate 3D kinematic estimation . To this end, we propose a novel approach for direct 3D human kinematic estimation D3KE from videos using deep neural networks. Our experiments demonstrate that the proposed end-to-end training is robust and outperforms 2D and 3D markerless motion capture based kinematic estimation pipelines in terms of joint angles error by a large margin (35% from 5.44 to 3.54 degrees). We show that D3KE is superior to the multi-step approach and can run at video framerate speeds. This technology shows the potential for clinical analysis from mobile devices in the future.

Characterization of the genetic landscape of Alzheimer’s disease (AD) and related dementias (ADD) provides a unique opportunity for a better understanding of the associated pathophysiological processes. We performed a two-stage
genome-wide association study totaling 111,326 clinically diagnosed/‘proxy’ AD cases and 677,663 controls. We found 75 risk loci, of which 42 were new at the time of analysis. Pathway enrichment analyses confirmed the involvement of amyloid/tau pathways and highlighted microglia implication. Gene prioritization in the new loci identified 31 genes that were suggestive of new genetically associated processes, including the tumor necrosis factor alpha pathway through the linear ubiquitin chain assembly complex. We also built a new genetic risk score associated with the risk of future AD/dementia or progression from mild cognitive impairment to AD/dementia. The improvement in prediction led to a 1.6- to 1.9-fold increase in AD risk from the lowest to the highest decile, in addition to effects of age and the APOE ε4 allele.