Endowing machines with social competence is not only a science fiction theme. It is also a long-held goal in computer science. Machines have changed how we work, communicate, and do art, science, and engineering, but they have had little effect on one of our core human needs: social interaction. Although digital communication has changed the way we interact with others, machines have arguably done little to enhance the quality of our face-to-face interactions and are seldom seen as tools to help us improve the way we interact with others. This is in part due to their lack of social competence thus far.

A crucial stepping stone towards social competence and the ability to display empathy is the ability to assess social experience. Social experience refers to internal states reflecting an individual's perception of a social situation, like enjoying a conversation or feeling attracted to someone they are interacting with. Social experience variables are hard to study because they are not directly observable and change over time. Researchers must rely on self-reports or third-party assessments (annotations). Algorithms for assessment of social experience generally take one of two approaches: 1) Direct modeling of the relationship between raw/derived signals and experience variables, utilizing sensor readings or outputs of detectors and feature extractors; and 2) intermediate modeling/detection of discrete actions performed during interactions (ie. speaking, laughter, gesturing).

In this thesis dissertation, we focus on in-the-wild mingling setting, where subjects are standing and are free to form and switch conversation groups as they desire. Data collection and annotation are paid special attention due to their relevance in a nascent field and the nuance involved in collecting and annotating social signals. Because the goal is to study machine social perception in real-life settings, interactions are not scripted and instrumentation is kept to a minimum.

We start with work concerned with the direct assessment of social experience, in this case of attraction, by exploring the predictive power of body acceleration. By analyzing accelerometer data from speed dating interactions, we investigate how the intensity and variations in body movement relate to self-reported attraction levels. This study sheds light on the predictive power of synchrony, mimicry, and convergence estimates for predicting attraction, and potentially other constructs related to affiliation.

We then address the detection of speaking, an action of wide interest in social signal processing due to the relevance of turn-taking in social experience. We address the limitations posed by visual cross-contamination in crowded mingling settings. We introduce a model that employs accelerometer readings and body poses to enhance the robustness of speaking status detection in a complex scene, with multiple interactions occurring simultaneously.

The dissertation also presents two novel datasets: ConfLab and REWIND, each serving a unique purpose. ConfLab, collected during a conference, is notable for its annotations of body joints, and improvements to the sensor setup resulting in increased data fidelity. Such methodological contributions to enable efficient and high-quality data collection are increasingly valuable given the scarcity of social interaction datasets, particularly in mingling settings. REWIND, gathered at a business networking event, stands out with its high-quality individual audio recordings, useful for the cross-modal study of multimodal signals such as speaking or laughter.

In a similar line, we present the Covfee software framework. Covfee challenges existing annotation methodologies by introducing and studying interfaces for continuous annotation for keypoints and actions. This framework was instrumental in efficiently processing the vast amounts of data collected in studies like ConfLab by streamlining the annotation process.

Also building on the Covfee framework, the dissertation culminates in an exploration of laughter annotation across different modalities. By comparing laughter annotations acquired in different conditions, the research highlights the complexities and nuances involved in interpreting social signals across different sensory inputs. We challenge the assumption that laughter intensity should be considered a property of the laughter episode. Instead, we find evidence that laughter evaluations differ significantly depending on the modalities available to the observer and that modalities with higher agreement will not necessarily result in the highest model performance. These results not only contribute to the study of laughter detection but also provide valuable insights for future research on multimodal social signal processing.

In summary, this dissertation weaves together a series of methodological contributions and novel findings, often derived from these new methods, each contributing to further our understanding of how to best train machines for social understanding and competence. ... Read more

Online experiments leveraging video conferencing offer significant advantages for studying human social interactions, including enhanced participant diversity and scalability. However, challenges include complex adjustments, privacy risks, software requirements, limited customization, and remote participant management. openVIMO is a software framework for creating video-based online interaction experiments built to address these challenges. It operates on open-source technologies, allowing deployment on researchers' servers without relying on third-party services. openVIMO facilitates the creation of web-based experiments with video calls, live monitoring of participants' progress or interactions, and comprehensive data collection, including audio and video. Moreover, it supports highly customizable experimental protocols and dynamic expansion by the research community. This tutorial will describe the functionality and design rationale underlying openVIMO. Moreover, it will outline possible application scenarios and provide examples for developing and managing studies using the platform. Finally, attendees will gain hands-on experience implementing a small-scale study of their design. ... Read more

Although laughter is known to be a multimodal signal, it is primarily annotated from audio. It is unclear how laughter labels may differ when annotated from modalities like video, which capture body movements and are relevant in in-the-wild studies. In this work we ask whether annotations of laughter are congruent across modalities, and compare the effect that labeling modality has on machine learning model performance. We compare annotations and models for laughter detection, intensity estimation, and segmentation, using a challenging in-the-wild conversational dataset with a variety of camera angles, noise conditions and voices. Our study with 48 annotators revealed evidence for incongruity in the perception of laughter and its intensity between modalities, mainly due to lower recall in the video condition. Our machine learning experiments compared the performance of modern unimodal and multi-modal models for different combinations of input modalities, training, and testing label modalities. In addition to the same input modalities rated by annotators (audio and video), we trained models with body acceleration inputs, robust to cross-contamination, occlusion and perspective differences. Our results show that performance of models with body movement inputs does not suffer when trained with video-acquired labels, despite their lower inter-rater agreement. ... Read more

Interpersonal attraction is known to motivate behavioral responses in the person experiencing this subjective phenomenon. Such responses may involve the imitation of behavior, as in mirroring or mimicry of postures or gestures, which have been found to be associated with the desire to be liked by an interlocutor. Speed dating provides a unique opportunity for the study of such behavioral manifestations of interpersonal attraction through the elimination of barriers to initiating communication, while maintaining significant ecological validity. In this paper we investigate the relationship between body movement, measured via accelerometer sensors, and self-reports or ratings of attraction and affiliation in a dataset of 399 speed dates between 72 subjects. Through machine learning experiments, we found that both features derived from a single individual's body movement and features designed to measure aspects of synchrony and convergence of the couple's body movement signals were predictive of different attraction ratings. Our statistical analysis revealed that the overall increase or decrease in an individual's body movement throughout an interaction is a potential indicator of friendly intentions, possibly related to the desire to affiliate. ... Read more

The benefits of exploiting multi-modality in the analysis of human-human social behaviour has been demonstrated widely in the community. An important aspect of this problem is the collection of data-sets that provide a rich and realistic representation of how people actually socialize with each other in real life. These subtle coordination patterns are influenced by individual beliefs, goals, and, desires related to what an individual stands to lose or gain in the activities they perform in their every day life. These conditions cannot be easily replicated in a lab setting and require a radical re-thinking of both how and what to collect. This tutorial provides a guide on how to create such multi-modal multi-sensor data sets when holistically considering the entire experimental design and data collection process. ... Read more

In this paper we introduce a novel method of estimating romantic, social and sexual attraction between two people by quantifying their bodily coordination using wearable sensors in a speed-date setting. We developed simple synchrony and convergence features, inspired from the literature and specifically adapted to be extracted from accelerometer data. To our knowledge, this is the first time that motion convergence is used for estimating attraction. Our features could predict one-way social attraction with a 73% Area under the ROC curve (AUC), out-performing previous work in a similar setting. We also showed that prediction performance increased when the male and female data are separated. We could also predict mutual romantic attraction with an AUC of 80%. Finally, we found that social attraction could be predicted better from movement correlation features whereas for romantic and sexual interest mimicry features were better indicators. Additionally, we found that 'mimicking of female to male' and 'convergence of female's movement to male's' were indicators of sexual and romantic mutual attraction in our data. ... Read more