Sensing and Modeling Human Behaviors In Complex Conversational Scenes

Sensing and Modeling Human Behaviors In Complex Conversational Scenes

Tan, S. (TU Delft Pattern Recognition and Bioinformatics)

Hung, H.S. (promotor)
Reinders, M.J.T. (promotor)
Tax, D.M.J. (copromotor)

Delft University of Technology

2023-05-03

Understanding human behavior has been an intriguing topic studied by many disciplines, including social science, neuroscience, etc. Humans exhibit social behaviors, through for example, interacting, conversing, empathizing with each other. Systematically and scientifically studying these behaviors often requires granular observations and measurements. With increasing digital sensor and computer sensing and processing capability, accurately measuring and recording large amount of real-life human social behavior has become possible. Computational methods, such as machine learning, can be developed to analyze these data in unprecedented ways by detecting and learning patterns in the signals. However, even with the available data and advanced machine learning methods, understanding human social behavior is still challenging, as it is contextual and could result in variations.

This thesis focuses on analyzing human behaviors in complex conversational scenes. It proposes novel computational methods that incorporate the context, which is the conversation group and the interaction scene. Prominent behavioral cues in social interaction include head and body orientations, as they are proxy indicators for visual attention and conversation group membership. This thesis first covers methods for head and body orientation estimation (under data-scarce and data-rich settings), and conversation group detection. These methods have an emphasis on learning from multimodal data and context modeling, and their efficacy is shown empirically. Then, the thesis addresses an open challenge in acquiring human social data in real-life by proposing an accurate and scalable method for data synchronization. Lastly, this thesis introduces a new dataset collected by the aforementioned synchronization method, capturing real-life interaction in a conference settings. Therein, results of tasks such as keypoint detection, action recognition, and conversation group detection are reported, which also motivate future research in this area. Combining these contributions in both computational method development and data collection, this thesis takes a step forward in understanding human behaviors in conversation scenes.

https://doi.org/10.4233/uuid:81a6854c-bc1d-4af5-816e-29a89ccc436d

Institutional Repository

doctoral thesis

© 2023 S. Tan