Exploring Retrospective Annotation in Long-videos for Emotion Recognition

Abstract

Emotion recognition systems are typically trained to classify a given psychophysiological state into emotion categories. Current platforms for emotion ground-truth collection show limitations for real-world scenarios of long-duration content (e.g. >10 minutes), namely: 1) Real-time annotation tools are distracting and become exhausting; 2) Perform retrospective annotation of the whole content in bulk (providing highly coarse annotations); or 3) Are used by external experts (depending on the number of annotators and their subjective experience). We explore a novel approach, the EmotiphAI Annotator, that allows undisturbed content visualisation and simplifies the annotation process by using segmentation algorithms that select brief clips for emotional annotation retrospectively. We compare three methods for content segmentation based on physiological data (Electrodermal Activity (EDA), emotion-based), scene (time-based), and random (control) selection. The EmotiphAI Annotator attained a B+ System Usability Scale score and low-average mental workload as per the NASA Task Load Index (40%). The reliability of the self-report was analysed by the inter-rater agreement (STD < 0.75), coherence across time segmentation methods (STD < 0.17), comparison against the state-of-the-art ground truth (STD < 0.7), and correlation to EDA (>0.3 to 0.8), where the EDA-based method obtained the overall best performance.