Beyond the Face: A Taxonomy of Situation Context Cues in Audio-Visual Emotion Perception

Abstract

Emotion recognition (ER) systems frequently fail in real-world settings because they are designed to be context-blind, relying solely on isolated facial or vocal expressions. While incorporating context is widely recognized as essential, the field lacks a structured, unified definition, often conflating situational factors with the target’s body language or the observer’s internal biases. This paper addresses this gap by establishing a comprehensive, non-overlapping taxonomy of situation context. Through a systematic literature review of empirical human studies, we disentangle situation context from observer and actor confounds, categorizing it into three distinct pillars: (1) the Objective Environment (physical and spatiotemporal surroundings), (2) the Social Array (the presence and actions of others), and (3) Semantic/Narrative Information (textual or verbal event descriptions). We map how cues within these categories systematically bias emotion judgments, such as shifting categorical labels or altering reaction times. By structuring the chaotic landscape of context, this taxonomy provides a concrete framework for designing context-aware, ecologically valid affective computing models.