Who, Where, How Matters

Evaluating Emotion Recognition under Context Shifts

Conference Paper (2026)
Author(s)

Sayak Mukherjee (TU Delft - Electrical Engineering, Mathematics and Computer Science)

Tom Viering (TU Delft - Electrical Engineering, Mathematics and Computer Science)

Bernd Dudzik (TU Delft - Electrical Engineering, Mathematics and Computer Science)

Research Group
Pattern Recognition and Bioinformatics
DOI related publication
https://doi.org/10.1145/3776574.3831130 Final published version
More Info
expand_more
Publication Year
2026
Language
English
Research Group
Pattern Recognition and Bioinformatics
Pages (from-to)
180-204
Publisher
ACM
ISBN (electronic)
9798400723186
Event
28th International Conference on Multimodal Interaction, ICMI 2026 (2026-10-05 - 2026-10-09), Napoli, Italy
Page Views
10
Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Abstract

Human emotion perception is inherently context-dependent, especially when observable behaviour is ambiguous. Vision-based emotion recognition systems learn to mimic emotion recognition from data. However, collecting a representative dataset is challenging due to the lack of consensus on what constitutes relevant contexts, as well as ethical and privacy constraints. This brings into question if such systems perform well across diverse real-world contexts. We consider three sources of context based on psychological theory: who expresses the emotion, where the emotion occurs, and how the data is collected. Our key contributions are: (a) Operationalising these contextual variations using EMOTIC, a widely used dataset; (b) Evaluating the supervised EMOTIC model when trained on one subgroup (e.g., women) and tested on another (e.g., men) within each contextual dimension (e.g. gender); and (c) Analysing the zero-shot performance of open-source (vision) language models under the same variations. Our results show that the EMOTIC model exhibits statistically significant performance gaps across contextual subgroups. We find large effect sizes for multiple metrics. These effects remain after controlling for dataset size, and some gaps even increase as the training sets grow. Zero-shot models underperform compared to EMOTIC, and also do not show consistent performance across contexts. These performance gaps are problematic for socially consequential settings, such as education, particularly regarding algorithmic discrimination and bias. Our work stresses the importance of context-aware data collection and evaluation for automated emotion recognition systems.