Collaborative partnerships are often formed following a first encounter. For example, unacquainted individuals may collaborate to complete a project, develop a product, or solve a problem. Using the Social Relations Model, this study examined the extent to which first-encounter trait perceptions predicted collaborative partner preferences. Previously-unacquainted participants (N = 297, 55 groups, 55.9% female) interacted dyadically and provided round-robin ratings of extraversion, honesty-humility, competence, and partner preference. At the target level, individuals who were consistently viewed as extraverted and competent were consistently preferred more as partners. At the relationship level, individuals who were uniquely viewed as honest-humble and competent were uniquely preferred more as partners. Findings underscore the relevance of target- and relationship-specific perceptions in predicting first-encounter collaborative partner preferences.

As intelligent technology and applications have become an integral part of nearly all aspects of people's daily lives, many intelligent systems have been designed to help people navigate the complex space of social interactions. One prominent strategy for such intelligent support is providing meaningful Ad Hoc Interventions (ADI), e.g., through timely notifications. An alternative is Technology-Supported Reflection (TSR), e.g., by offering information about activities in one's past for personal insights. In contrast to straight-up interventions, the aim of the latter strategy is not to directly augment human skills but instead support learning and personal growth over time. However, while TSR has seen widespread interest in applications in some areas, such as physical fitness and mental health, its use for improving human social interactions has not yet been systematically explored. Concretely, it is currently unclear 1) what forms of self-reflection systems intend to support, 2) how their different technological components (e.g., data collection, information integration) are involved in providing support, and 3) what common limitations and design challenges they face. In this article, we present the results of a systematic literature review focusing on these questions to provide a structured foundation for targeted research. Concretely, we identified and analysed a collection of 23 relevant papers, each describing a system deploying TSR to support humans with elements of social interactions.We constructed a framework with a set of features to comprehensively describe and analyze the systems that support self-reflection, including their application domains, how they fit into the existing design framework, how they facilitate learning through reflection, how adaptive they are to individual users, and how they were evaluated. Finally, we propose a direction for designing systems that support individual's social interactions through self-reflection in an adaptive manner.

How people evaluate, select, and engage with others in cooperative settings significantly impacts their well-being, happiness, and success. However, navigating these processes is complex. Equipping systems with the ability to recognize, interpret, and even engage during such socio-cognitive processes can increase their potential to support humans in these socio-cognitive processes and be more successful in adjusting to the social environment they are embedded in (e.g., understanding human preferences and attitudes), leading to better quality interactions and decision-making for future partners. Yet, the developments of such systems depend on available datasets. However, based on our knowledge, no dataset exists that can be used to model partner selection for joint tasks. To support research focused on creating such intelligent systems, we introduce the PARSEL dataset – a comprehensive corpus of dyadic interactions designed for computational modeling of PARtner SELection processes and collaborative behavior. In total, 297 participants took part in the datasets. The dataset contains measurements of partner selection decisions over three different stages, as well as factors that may influence partner selection in the context of (online) social interactions. It includes audiovisual recordings that offer fine-grained behavioral cues used during these interactions, self-reported traits, and reported perceptions of person-, situation- and team-specific phenomena. By providing this resource, we aim to foster advancements in computational methods that can effectively model and augment socio-cognitive processes, contributing to socially aware intelligent systems and enhanced human-system interactions.

Social signal processing develops automated approaches to detect, analyze, and synthesize social signals in human–human as well as human–machine interactions by means of machine learning and sensor data processing. Most works analyze individual or dyadic behavior, while the analysis of group or team interactions remains limited. We present a case study of an interdisciplinary work process for social signal processing that can develop automatized measures of complex team interaction dynamics, using team task and social cohesion as an example. In a field sample of 25 real project team meetings, we obtained sensor data from cameras, microphones, and a smart ID badge measuring acceleration. We demonstrate how fine-grained behavioral expressions of task and social cohesion in team meetings can be extracted and processed from sensor data by capturing dyadic coordination patterns that are then aggregated to the team level. The extracted patterns act as proxies for behavioral synchrony and mimicry of speech and body behavior which map onto verbal expressions of task and social cohesion in the observed team meetings. We reflect on opportunities for future interdisciplinary or collaboration that can move beyond a simple producer–consumer model.

Free-standing social conversations constitute a yet underexplored setting for human behavior forecasting. While the task of predicting pedestrian trajectories has received much recent attention, an intrinsic difference between these settings is how groups form and disband. Evidence from social psychology suggests that group members in a conversation explicitly self-organize to sustain the interaction by adapting to one another’s behaviors. Crucially, the same individual is unlikely to adapt similarly across different groups; contextual factors such as perceived relationships, attraction, rapport, etc., influence the entire spectrum of participants’ behaviors. A question arises: how can we jointly forecast the mutually dependent futures of conversation partners by modeling the dynamics unique to every group? In this paper, we propose the Social Process (SP) models, taking a novel meta-learning and stochastic perspective of group dynamics. Training group-specific forecasting models hinders generalization to unseen groups and is challenging given limited conversation data. In contrast, our SP models treat interaction sequences from a single group as a meta-dataset: we condition forecasts for a sequence from a given group on other observed-future sequence pairs from the same group. In this way, an SP model learns to adapt its forecasts to the unique dynamics of the interacting partners, generalizing to unseen groups in a data-efficient manner. Additionally, we first rethink the task formulation itself, motivating task requirements from social science literature that prior formulations have overlooked. For our formulation of Social Cue Forecasting, we evaluate the empirical performance of our SP models against both non-meta-learning and meta-learning approaches with similar assumptions. The SP models yield improved performance on synthetic and real-world behavior datasets.

Although laughter is known to be a multimodal signal, it is primarily annotated from audio. It is unclear how laughter labels may differ when annotated from modalities like video, which capture body movements and are relevant in in-the-wild studies. In this work we ask whether annotations of laughter are congruent across modalities, and compare the effect that labeling modality has on machine learning model performance. We compare annotations and models for laughter detection, intensity estimation, and segmentation, using a challenging in-the-wild conversational dataset with a variety of camera angles, noise conditions and voices. Our study with 48 annotators revealed evidence for incongruity in the perception of laughter and its intensity between modalities, mainly due to lower recall in the video condition. Our machine learning experiments compared the performance of modern unimodal and multi-modal models for different combinations of input modalities, training, and testing label modalities. In addition to the same input modalities rated by annotators (audio and video), we trained models with body acceleration inputs, robust to cross-contamination, occlusion and perspective differences. Our results show that performance of models with body movement inputs does not suffer when trained with video-acquired labels, despite their lower inter-rater agreement.

In this article we introduce Mementos: the first multimodal corpus for computational modeling of affect and memory processing in response to video content. It was collected online via crowdsourcing and captures 1995 individual responses collected from 297 unique viewers responding to 42 different segments of music videos. Apart from webcam recordings of their upper-body behavior (totaling 2012 minutes) and self-reports of their emotional experience, it contains detailed descriptions of the occurrence and content of 989 personal memories triggered by the video content. Finally, the dataset includes self-report measures related to individual differences in participants' background and situation (Demographics, Personality, and Mood), thereby facilitating the exploration of important contextual factors in research using the dataset. We describe 1) the construction and contents of the corpus itself, 2) analyse the validity of its content by investigating biases and consistency with existing research on affect and memory processing, 3) review previously published work that demonstrates the usefulness of the multimodal data in the corpus for research on automated detection and prediction tasks, and 4) provide suggestions for how the dataset can be used in future research on modeling Video-Induced Emotions, Memory-Associated Affect, and Memory Evocation.

Interpersonal attraction is known to motivate behavioral responses in the person experiencing this subjective phenomenon. Such responses may involve the imitation of behavior, as in mirroring or mimicry of postures or gestures, which have been found to be associated with the desire to be liked by an interlocutor. Speed dating provides a unique opportunity for the study of such behavioral manifestations of interpersonal attraction through the elimination of barriers to initiating communication, while maintaining significant ecological validity. In this paper we investigate the relationship between body movement, measured via accelerometer sensors, and self-reports or ratings of attraction and affiliation in a dataset of 399 speed dates between 72 subjects. Through machine learning experiments, we found that both features derived from a single individual's body movement and features designed to measure aspects of synchrony and convergence of the couple's body movement signals were predictive of different attraction ratings. Our statistical analysis revealed that the overall increase or decrease in an individual's body movement throughout an interaction is a potential indicator of friendly intentions, possibly related to the desire to affiliate.

Space exploration is evolving with the recent increase in interest and investment. For the success of planned long-duration crewed missions, good interpersonal interactions between crew members are crucial. In this study, we evaluate the use of wearables for detection and estimation of the quality of each social interaction participants have throughout a long mission rather than aggregate measures of interactions. Our proposed method utilizes Temporal Convolutional Networks(TCNs) for extracting individual representations from acceleration and audio streams and learnable pooling layers(NetVLAD) to aggregate these representations into fixed-size representations. Use of NetVLAD layers provides an intelligent alternative to simple aggregation for handling variable-sized interactions and interactions with missing data. We evaluate our method on a 4-month simulated space mission where 5 participants wore Sociometric Badges and provided reports on their interactions in terms of effectiveness, frustration, and satisfaction. Our method provides an average ROC-AUC score of 0.64. Since we are not aware of any comparable baselines, we compare our method to hand-crafted features formerly utilized for cohesion estimation in similar scenarios and show it significantly outperforms them. We also present ablation studies where we replace the components in our approach with well-known alternatives and show that they provide better performance than their respective counterparts.

Intelligent systems might benefit from automatically detecting when a stimulus has triggered a user's recollection of personal memories, e.g., to identify that a piece of media content holds personal significance for them. While computational research has demonstrated the potential to identify related states based on facial behavior (e.g., mind-wandering), the automatic detection of spontaneous recollections specifically has not been investigated this far. Motivated by this, we present machine learning experiments exploring the feasibility of detecting whether a video clip has triggered personal memories in a viewer based on the analysis of their Head Rotation, Head Position, Eye Gaze, and Facial Expressions. Concretely, we introduce an approach for automatic detection and evaluate its potential for predictions using in-the-wild webcam recordings. Overall, our findings demonstrate the capacity for above chance detections in both settings, with substantially better performance for the video-independent variant. Beyond this, we investigate the role of person-specific recollection biases for predictions of our video-independent models and the importance of specific modalities of facial behavior. Finally, we discuss the implications of our findings for detecting recollections and user-modeling in adaptive systems.

In this work, we propose an approach for detecting conversation groups in social scenarios like cocktail parties and networking events, from overhead camera recordings. We posit the detection of conversation groups as a learning problem that could benefit from leveraging the spatial context of the surroundings, and the inherent temporal context in interpersonal dynamics which is reflected in the temporal dynamics in human behavior signals, an aspect that has not been addressed in recent prior works. This motivates our approach which consists of a dynamic LSTM-based deep learning model that predicts continuous pairwise affinity values indicating how likely two people are in the same conversation group. These affinity values are also continuous in time, since relationships and group membership do not occur instantaneously, even though the ground truths of group membership are binary. Using the predicted affinity values, we apply a graph clustering method based on Dominant Set extraction to identify the conversation groups. We benchmark the proposed method against established methods on multiple social interaction datasets. Our results showed that the proposed method improves group detection performance in data that has more temporal granularity in conversation group labels. Additionally, we provide an analysis in the predicted affinity values in relation to the conversation group detection. Finally, we demonstrate the usability of the predicted affinity values in a forecasting framework to predict group membership for a given forecast horizon.

This paper focuses on the automatic classification of self-assessed personality traits from the HEXACO inventory during crowded mingle scenarios. These scenarios provide rich study cases for social behavior analysis but are also challenging to analyze automatically as people in them interact dynamically and freely in an in-the-wild face-to-face setting. To do so, we leverage the use of wearable sensors recording acceleration and proximity, and video from overhead cameras. We use 3 different behavioral modality types (movement, speech and proximity) coming from 2 sensors (wearable and camera). Unlike other works, we extract an individual's speaking status from a single body worn triaxial accelerometer instead of audio, which scales easily to large populations. Additionally, we study the effect of different combinations of modality types on the personality estimation, and how this relates to the nature of each trait. We also include an analysis of feature complementarity and an evaluation of feature importance for the classification, showing that combining complementary modality types further improves the classification performance. We estimate the self-assessed personality traits both using a binary classification (community's standard) and as a regression over the trait scores. Finally, we analyze the impact of the accuracy of the speech detection on the overall performance of the personality estimation.

We are organizing again the workshop on Interdisciplinary Insights into Group and Team Dynamics which is a joint effort between researchers in the the ICMI and INGRoup (Interdisciplinary Network for Group Research) communities. This workshop aims to provide a common destination for researchers to exchange ideas and collaborate. We have found in previous years that instigating interdisciplinary collaborations can be hard. The aim of this workshop is to sustain a joint community to foster continued cross-disciplinary exchange and mutual understanding.

Human head orientation estimation has been of interest because head orientation serves as a cue to directed social attention. Most existing approaches rely on visual and high-fidelity sensor inputs and deep learning strategies that do not consider the social context of unstructured and crowded mingling scenarios. We show that alternative inputs, like speaking status, body location, orientation, and acceleration contribute towards head orientation estimation. These are especially useful in crowded and in-the-wild settings where visual features are either uninformative due to occlusions or prohibitive to acquire due to physical space limitations and concerns of ecological validity. We argue that head orientation estimation in such social settings needs to account for the physically evolving interaction space formed by all the individuals in the group. To this end, we propose an LSTM-based head orientation estimation method that combines the hidden representations of the group members. Our framework jointly predicts head orientations of all group members and is applicable to groups of different sizes. We explain the contribution of different modalities to model performance in head orientation estimation. The proposed model outperforms baseline methods that do not explicitly consider the group context, and generalizes to an unseen dataset from a different social event.

Enabling computer-based applications to display intelligent behavior in complex social settings requires them to relate to important aspects of how humans experience and understand such situations. One crucial driver of peoples' social behavior during an interaction is the interdependence they perceive, i.e., how the outcome of an interaction is determined by their own and others' actions. According to psychological studies, both the nonverbal behavior displayed by Motivated by this, we present a series of experiments to automatically recognize interdependence perceptions in dyadic face-to-face negotiations using these sources. Concretely, our approach draws on a combination of features describing individuals' Facial, Upper Body, and Vocal Behavior with state-of-the-art algorithms for multivariate time series classification. Our findings demonstrate that differences in some types of interdependence perceptions can be detected through the automatic analysis of nonverbal behaviors. We discuss implications for developing socially intelligent systems and opportunities for future research.