Automatic Smile and Frown Recognition with Kinetic Earables

Conference paper (2019)

Authors

Seungchul Lee Korea Advanced Institute of Science and Technology
Chulhong Min Nokia Bell Labs
Alessandro Montanari Nokia Bell Labs
Akhil Mathur Nokia Bell Labs, University College London
Youngjae Chang Korea Advanced Institute of Science and Technology
Junehwa Song Korea Advanced Institute of Science and Technology
F. Kawsar Internet of Things - IDE , Nokia Bell Labs
Research Group
Internet of Things (IDE) (TU Delft)
Kinetic modeling Earable Facs Smile and frown recognition
More Info
expand_more

Published Date

2019

Language

English

Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Faculty

Industrial Design Engineering

Department

Sustainable Design Engineering

Research Group

Internet of Things

Abstract

In this paper, we introduce inertial signals obtained from an earable placed in the ear canal as a new compelling sensing modality for recognising two key facial expressions: Smile and frown. Borrowing principles from Facial Action Coding Systems, we first demonstrate that an inertial measurement unit of an earable can capture facial muscle deformation activated by a set of temporal microexpressions. Building on these observations, we then present three different learning schemes - shallow models with statistical features, hidden Markov model, and deep neural networks to automatically recognise smile and frown expressions from inertial signals. The experimental results show that in controlled non-conversational settings, we can identify smile and frown with high accuracy (F1 score: 0.85).