On Fine-grained Temporal Emotion Recognition in Video: How to Trade off Recognition Accuracy with Annotation Complexity?

Fine-grained emotion recognition is the process of automatically identifying the emotions of users at a fine granularity level, typically in the time intervals of 0.5s to 4s according to the expected duration of emotions. Previous work mainly focused on developing algorithms to recognize only one emotion for a video based on the user feedback...

A Cross-Corpus Speech-Based Analysis of Escalating Negative Interactions

The monitoring of an escalating negative interaction has several benefits, particularly in security, (mental) health, and group management. The speech signal is particularly suited to this, as aspects of escalation, including emotional arousal, are proven to easily be captured by the audio signal. A challenge of applying trained systems in real...

Investigating Affective Responses toward In-Video Pedestrian Crossing Actions using Camera and Physiological Sensors

Automatically inferring drivers' emotions during driver-pedestrian interactions to improve road safety remains a challenge for designing in-vehicle, empathic interfaces. To that end, we carried out a lab-based study using a combination of camera and physiological sensors. We collected participants' (N=21) real-time, affective (emotion self...

Towards Artificial Empathic Memory: Accounting for the Influence of Personal Memories in Automatic Predictions of Affect

Super-resolution to enhance low-resolution thermal facial expression images for thermal facial emotion recognition

Facial emotion recognition from thermal images has gained more attention in recent years. Thermal cameras capture the heat emitted by objects and therefore thermal images are not sensitive to illumination changes. Furthermore, changes in temperature can indicate emotions and it is harder for humans to fake emotions in front of a thermal camera....

Corrnet: Fine-grained emotion recognition for video watching using wearable physiological sensors

Recognizing user emotions while they watch short-form videos anytime and anywhere is essential for facilitating video content customization and personalization. However, most works either classify a single emotion per video stimuli, or are restricted to static, desktop environments. To address this, we propose a correlation-based emotion...

Exploring Personal Memories and Video Content as Context for Facial Behavior in Predictions of Video-Induced Emotions

Empirical evidence suggests that the emotional meaning of facial behavior in isolation is often ambiguous in real-world conditions. While humans complement interpretations of others' faces with additional reasoning about context, automated approaches rarely display such context-sensitivity. Empirical findings indicate that the personal...

ET-CycleGAN: Generating thermal images from images in the visible spectrum for facial emotion recognition

Facial thermal imaging has in recent years shown to be an efficient modality for facial emotion recognition. However, the use of deep learning in this field is still not fully exploited given the small number and size of the current datasets. The goal of this work is to improve the performance of the existing deep networks in thermal facial...

Creating a Mood Database for automated affect analysis

Affect-adaptive systems are dependent on their ability to automatically recognize a user’s affective state. This study aims to contribute to the creation of an affect-adaptive system that can recognize negative moods of elderly in care homes from a video feed, and improve it by adapting the lighting in the room. An affective database of videos...

Automatically predicting mood from expressed emotions

Affect-adaptive systems have the potential to assist users that experience systematically negative moods. This thesis aims at building a platform for predicting automatically a person’s mood from his/her visual expressions. The key word is mood, namely a relatively long-term, stable and diffused affective state, as opposed to the short-term,...

Multimodal recognition of emotions

This thesis proposes algorithms and techniques to be used for automatic recognition of six prototypic emotion categories by computer programs, based on the recognition of facial expressions and emotion patterns in voice. Considering the applicability in real-life conditions, the research is carried in the context of devising person independent...

Automatic Emotion Analysis Based on Speech

Extracting emotions from face-to-face communication

In our life we get more and more dependent on our computer and we have less time for face-to-face social activities with friends and families. In face?to?face communication our faces convey lots of emotions using facial expressions and lots of information is transmitted faster non-verbally than verbally, through the facial expressions. Using...

Optical flow-based facial feature tracking to recognize AUs modeled by Bayesian networks

This thesis describes a Bayesian Network (BN) model for recognizing the “Action Units (AUs)” of a facial expression using video sequence images as input. Features were extracted by using an optimal estimation optical flow method coupled with a physical (muscle) model describing the facial structure. The muscle action patterns are used for...

Intelligent methods for automated video surveillance

At the Man Machine Interaction research group at the Delft University of Technology research is being done on the subject of aggression detection in trains. The goal of this project is to research different aspects of train surveillance, including video surveillance, but also audio surveillance storyboard based modeling. This thesis discusses...

The recognition of emotions from speech using GentleBoost classifier: A comparison approach

The recognition of the internal emotional state of one person plays an important role in several human-related fields. Among them, human-computer interaction has recently received special attention. The current research is aimed at the analysis of segmentation methods and of the performance of the GentleBoost classifier on emotion recognition...