Nuances of Interrater Agreement on Automatic Affect Prediction from Physiological Signals
A Systematic Review of Datasets Presenting Various Agreement Measures and Affect Representation Schemes
O. Fron (TU Delft - Electrical Engineering, Mathematics and Computer Science)
Bernd Dudzik – Mentor (TU Delft - Pattern Recognition and Bioinformatics)
Catherine Oertel – Graduation committee member (TU Delft - Interactive Intelligence)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
This study explores the influence of interrater agreement measures and affect representation schemes in automatic affect prediction systems using physiological signals. These systems often use supervised learning and require unambiguous and objective labeling, a challenge when multiple human annotators are involved, which can affect model performance.
The research involved the first part of a two-stage process: systematically reviewing datasets and their characteristics concerning interrater agreement on the affective interpretation of physiological signals. This stage established a reliable foundation for the second step: a future analysis of model performance reported in technical papers utilizing these datasets. The main takeaways were that the number of raters varies significantly over datasets and the complexity introduced by combining affect representation schemes can negatively affect interrater agreement.