Interpreting and managing social interactions is vital for social well-being, yet existing technologies fall short, particularly in group settings. This research aims to develop advanced machine perception systems for Social Signal Processing to accurately model human social behavior. Our multi-modal generative model aims to integrate multi-modal sensory data input data, contextual information and subjective observers’ narratives, utilizing them as complex input to an adapted Large Language Model, and producing plausible narratives that refect various human perspectives. This human-centered approach leverages both low-level cues and high-order events, ensuring adaptability to diverse observers and contexts. The model’s potential areas of application include cross-cultural interactions, social group integration, and professional meetings, enhancing social harmony and productivity.