Early detection of depression is crucial in mental healthcare. Augmenting depression diagnosing with AI seems to be promising in detecting depression from subtle non-verbal cues and early signs that can be missed from domain experts. For this to be achieved, AI procedures and decision processes need to be interpretable to humans. In this thesis, we use and evaluate a saliency-based explainability framework for a multi-modal depression-prediction model and validate its outputs through human judgment. The multimodal input is created by combining high level facial features extracted from Action-Unit via a 1D-CNN and high level vocal features extracted from log-mel spectrograms via a modified AlexNet. Then a simple Feed Forward Network is used as a classification predictor for 3.5 second segments.

To assess whether these AI-flagged moments align with human reasoning, 17 lay participants viewed thirty 8.5-second clips (half depressed, half non-depressed). For each clip they (1) rated depression confidence on a 1–10 scale, (2) selected the single frame they found most influential, and (3) described the facial or vocal cues that informed their choice. The goal is for the participants to give us an insight on what the model may be 'seeing'. So we ask them to tell us what facial and voice features they observed in their influential moments. From those experiments, we gained some useful insights to the model. The results show that participants observations in non-verbal cues are valuable and align with literature findings. And we find that there is alignment in participant's observations on their own influential moment and on the model's salient moment when the salient moment is correctly classified by the model and they do not align when the salient moment is wrongly classified by the model. These findings suggest that humans and the model value similar cues to make the correctly classify depression, and help to enhance the interpretability of AI models.

The emergence of Language Language Models (LLMs)-based agents represents a significant advancement in artificial intelligence (AI), offering new possibilities for complex problem-solving and interaction within a virtual environment. Our work is based on the Voyager paper [1], which is a state-of-the-art LLM-based agent for Minecraft. However, this system suffers from some significant limitations, such as its reliance on closed-source LLMs and lack of social awareness. Indeed, current open-source LLMs often fail to match closed-source ones in the agent setting, leaving research reliant on third-party closed-source technology [1] [2]. This gap highlights the need for alternative strategies to enhance LLM performance without the high costs associated with fine-tuning. To address these challenges, we propose the Collaborative Voyager, a new architecture designed to enable agent collaboration and social awareness using open-source LLMs. Inspired by the social intelligence hypothesis, which suggests that intelligence emerges from social interactions, we propose collaboration as an alternative learning paradigm for LLMs. This alternative learning paradigm could potentially supplement the expensive fine-tuning currently needed to bridge the performance gap between open-source and closed-source models in the agent setting [2]. Our approach involves developing a framework that allows agents to communicate, understand, and learn from each other, enabling them to correct errors and adapt to new tasks dynamically. By using a memory module, our agent is able to remember interactions and learn from them in order to accomplish a task that it was previously unable to do on its own. Through various experiments, we demonstrate that collaboration significantly enhances the performance of LLM agents in both task completion and adaptability, addressing issues like hallucinations. This study provides insights into developing more sophisticated, adaptable AI systems capable of dynamic interactions and problem-solving. These findings have potential applications extending beyond Minecraft and virtual environments to fields such as robotics, where collaboration and social awareness are crucial.

In the future, autonomous social robots are expected to seamlessly integrate into our society. To be perceived as interactive partners rather than mere tools, these robots must be embodied and capable of navigating complex, dynamic environments. This study explores the critical role of embodiment and examines the application of human-inspired socio-cognitive mechanisms in embodied agents. We conducted a systematic scoping review of 36 publications sourced from Scopus, IEEE Xplore, Web of Science, and PubMed. Our findings reveal a diverse array of human-inspired implementations in embodied agents, including curiosity-driven learning, categorical thinking, and predictive learning. Notably, human-like eye-gaze plays a crucial role in enhancing the anthropomorphism of these agents. By drawing inspiration from human interactions, we can transition from pre-programmed robots to fully autonomous agents that exhibit emergent behaviours and adapt to ever-changing conditions. Embodiment allows agents to communicate their intentions and desires through various modes, facilitating richer interactions. Additionally, multiple studies underscore the importance of using embodied agents to study human behaviours. Furthermore, the choice of embodiment type must depend on the task at hand; for some tasks, virtual embodiment is more effective, while for others, physical embodiment is preferable. This review provides a comprehensive starting point for future researchers interested in developing human-inspired embodied agents.

Continual learning (CL) enables intelligent systems to continually acquire, adapt, and apply knowledge, representing a dynamic paradigm in AI. For embodied agents—interacting with their environment physically and cognitively—CL enhances adaptability and reduces training costs significantly. In this literature review, we contribute by focusing on the application of CL in such agents, showcasing the approaches, means of evaluation and practical uses of this cognitive framework in real-world scenarios. We conclude that while CL holds promise for embodied agents, there exists a notable gap between the theoretical evaluation of CL and the complex real-world scenarios these agents operate in.

Virtual agents have demonstrated remarkable progress in both competitive and cooperative en- vironments. Embodied agents, which enhance AI interactions with the physical world, show great promise for a variety of use cases in both virtual and non-virtual settings. This literature review ex- amines the intersection of embodied virtual agents with cognitive and social frameworks derived from human behavior, focusing on open-ended learning and intrinsic motivation. These paradigms, inspired by human learning and adaptability, offer a path to- wards addressing the limitations of current artificial systems. The literature survey provides a thorough analysis of the research landscape, discussing the definitions, applications, and benefits of embodied agents in virtual settings. Furthermore, it evaluates the methods and benchmarks used to assess the ca- pabilities of these agents, while offering possible solutions for developing the next generation of embodied agents.

This research paper aims to present how Theory of Mind (ToM) - the ability that allows humans to attribute mental states to others - can be used in the context of physically and virtually embodied computational agents. The focus is on using ToM for perspective-taking in environments with multiple computational agents interacting together. A Systematic Literature Review (SLR) was conducted providing a structured search process for collecting the literature. The findings from this review were obtained from 38 selected papers found in the literature since 2018. The review presented how computational ToM was inspired by the human ToM. Further, it summarised the current applications for perspective-taking for multi-agent settings, and it collected insights on high-level implementations of ToM agents. The findings highlight the complexity of ToM agents and the importance of ToM for agent interactions.

Active inference is a theory of the human brain characterising behaviour that minimises surprise. The free energy principle accounts for the adaptive behaviours of organisms through action, perception, and learning aimed at optimising reward or surprise. This study systematically reviews relevant literature to address their methodologies, relevance to mimicking social human behaviour, challenges, and limitations to guide future research by succinctly reporting previous findings and research gaps. Active inference models are extended with deep active inference, free-energy models, multimodal deep belief networks, predictive coding, and probabilistic programming. These models employ goal-directed, epistemic, reward-seeking, and decision-making behaviours and simulate cumulative culture. However, some of these models do not translate well to complex real-life applications due to their simplicity, computational demands, or the assumptions upon which they are based. Challenges with real-life applications include difficulty scaling to high-dimensional data and model simplicity. Furthermore, some experiments did not have enough data to validate or train their models.

Recovering the appearance and physical parameters of elastic objects from multi-view video is essential for many applications that require simulation of the real world. Past methods for this task have provided accurate results in recovering physical properties; however, their re- liance on Neural Radiance Fields (NeRFs) for novel view synthesis means they trade off visual quality for rendering speed. To address this issue, we present a novel framework for the joint appearance reconstruction and physical parameter estimation of elastic objects relying on 3D Gaussian splatting. Our key insight is that dynamic 3D Gaussian kernels extracted from multi-view video can be used to reconstruct the object’s geometry and supervise elastic parame- ter fitting through a differentiable physics engine. Novel views and object behaviours can then be constructed by forward simulating the extracted mesh and using it to drive the Gaussian kernels. We demon- strate that our method is competitive with the state of the art on phys- ical parameter estimation while being better at reconstructing object appearance. Additionally, our method can simulate novel views and object interactions at near real-time rates that outperform past ap- proaches.

Human Theory of Mind (ToM), the ability to infer others’ mental states, is essential for effective social interaction. It allows us to predict behavior and make decisions accordingly.In Human Robot Interaction (HRI), however, this remains a significant challenge, especially in dynamic, real-world scenarios. Enabling robots to possess ToM-like capabilities has the potential to greatly improve their interaction with humans. Recent advancements have introduced Large Language Models (LLMs) as robot controllers, leveraging their strengths in generalization, reasoning, and code comprehension. Some have claimed that LLMs may exhibit emergent ToM capabilities, but these claims have yet to be substantiated with rigorous evidence. This study investigates the ToM-like abilities of Multimodal Large Language Models (MLLMs) by creating a benchmark dataset from humans performing object rearrangement tasks in a simulated environment. The dataset visually captures the participants’ behavior and textually captures their internal monologues. Based on this dataset (text, video, or hybrid) three state-of-the-art models made predictions about the participants’ belief updates. While the results do not conclusively establish ToM capabilities in MLLMs, they offer promising insights into mental model inference and suggest future directions for research in this domain.

Images possess the ability to convey a wide range of emotions, and extracting affective information from images is crucial for affect prediction systems. This process can be achieved through the application of machine learning algorithms. Categorical Emotion States (CES) and Dimensional Emotion Space (DES) are two typical models used for representing emotions. Moreover, the development of a mapping schema between these representations can contribute to benefit the research in AI and psychology. Consequently, this study focuses on investigating the feasibility of translating emotions from DES to CES. To accomplish this goal, relevant databases are identified and combined as the training data, and machine learning models, namely Naive Bayes, K-nearest Neighbors, and Decision Tree are employed to perform the classification task. The results indicate the superior performance of the K-nearest Neighbors classifier, exhibiting higher mean accuracy (60%) and low standard deviation (0.004) among all implemented classifiers. Overall, the translation from DES to CES offers several advantages, including a simplified and interpretable representation of emotions, as well as the provision of a common language for discussing and expressing emotions.

Studies in Music Affect Content Analysis use varying emotion schemes to represent the states induced when listening to music. However, there are limited studies that explore the translation between these representation schemes. This paper explores the feasibility of using machine learning models to translate from a dimensional scheme of Valence, Energy and Tension, to a categorical emotion of Anger, Fear, Happy, Sad, or Tender, specifically in the context of musical stimuli. Additionally, this paper considers how the close proximity of certain emotions in the dimensional space, such as Fear and Anger, negatively influence the performance of translation models. This paper reflects on past studies and presents new results concluding feasible translations of music affect content, moreover, providing suggestions for future analysis.

There is a correlation between music and affect which researchers try to define and use in technology to improve healthcare and users' experience in music-related technology. However, since affect is a complex term there is not a specified way on how to represent different affective states in systems. A systematic literature review was performed to give an overview of affect representation schemes (ARS) used in music affect prediction. All the studies included in this survey were found in Scopus, Web of Science and IEEE Explorer and focus on automatic affect prediction in music. What is more, the literature written in a different language than English and not from the Computer Science field was excluded. Considering the time constraint of 10 weeks, an additional feasibility filtering was conducted. That is only the documents which mention common benchmark datasets for this task were included. The datasets were chosen after the research of related work and different literature reviews related to this topic. We were aware that it might include a bias correlated to dataset popularity. At the final stage of filtering, 113 records were chosen for data extraction from which 51 were used in the analysis due to time constraints and a single person conducting this research. After synthesising and analysing the extracted data we observed that the schemes can be distinguished between dimensional and categorical approaches and both are similarly popular. A rare but existing ARS are developed by combining both scheme types in a unique representation. What is more, there is no easily visible trend over time in the usage of certain schemes and only 66% of the reviewed studies support their choices with psychological theories. While conducting this research, we encountered limitations in terms of time and the research group, so we leave a gap for future improvements in this project.

Affective Video Content Analysis aims to automatically analyze the intensity and type of affect (emotion or feeling) that are contained in a video and are expected to arise in users while watching that video. This study aims to provide a systematic overview of various affect representation schemes utilised by researchers in the field of Affective Video Content Analysis and the reasons behind their choice. The main objectives of the study were to investigate the diversity of affect representation scheme types, their popularity over time, the basis of their selection, and the relationship between input data sources, in terms of direct and implicit analysis, and scheme types. Following the PRISMA guidelines to conduct a systematic literature review, a total of 45 papers were included in the study which were original journals and conference proceedings in the field of Affective Video Content Analysis and that were related only to Affective Movie Content Analysis published in English after 2008. Papers concerning Video Emotion Recognition were excluded from the review. The findings reveal that dimensional, categorical, and combined approaches are commonly used in this field, with the dimensional approach based on valence and arousal being the most prevalent. However, there is no significant trend in the popularity of affect representation scheme types over time. The study highlights the lack of clear motivation and explicit justifications for scheme selection, emphasizing the need for transparency and the inclusion of psychological theories as a basis for scheme choices. Additionally, the study found that audiovisual data was the most commonly used input compared to physiological signals and visible behavioural data. The study acknowledges its limitations, including time constraints and single researcher involvement, and suggests allocating more time and involving multidisciplinary teams for comprehensive insights.

Physiological signals, such as Electroencephalogram (EEG), Glavic Skin Response (GSR), or Body Temperature, are common inputs for Automatic Affect Recognition (AAR) systems. One of the crucial elements of AAR is the Affect Representation Scheme (ARS) used to define the affective states recognized by the system (e.g., happiness, sadness, fear, anger). Throughout the years many AAR reviews have been published. However, most of them do not include a detailed analysis of ARSs and the motivation behind them. This paper aims to fill this knowledge gap by performing a Systematic Literature Review (SLR) of Computer Science papers that propose a Physiological-signal-based AAR (PAAR) system. We explore how researchers discuss and choose an ARS and whether they base it on actual psychological theories. Eligible papers are retrieved from 4 databases: Web Of Science, IEEExplore, Scopus, and ACM Digital Library. Due to time limitations, the review is done rapidly and some additional search constraints are applied for feasibility. The most significant restrictions are: considering papers published since 2020 and performing experiments on specific benchmarking datasets. We take these constraints and their possible impact into consideration when interpreting the results. The presented review procedure can be stripped from the additional filters and reused for a full review. In total 115 papers are processed. The majority of papers introduce an EEG-based emotion recognition system. The analysis of the extracted information reveals that dimensional ARSs, in particular, Valence/Arousal model is the most popular. Moreover, authors often choose to reduce the dimensions to high/low values. Categorical ARSs are less frequent and usually are adopted from the dataset. Lastly, the authors do not provide extensive motivation for the choice of ARS and rarely refer to psychological theories.

Automatic affect prediction systems usually assume its underlying affect representation scheme (ARS). This systematic review aims to explore how different ARS are used for in affect prediction systems based on spoken input. The focus is only on the audio input from speakers. Various datasets for speech emotion recognition were also involved in the study to understand the motivation for certain (categorical or dimensional) schemes used for emotions. The basis, popularity, advantages and target affective states were investigated. We used Scopus and Web of Science to extract the papers, focusing on the systems in the field of Computer Science in English language. In summary, our exploration of affect representation schemes in Speech Emotion Recognition (SER) reveals a predominant focus on categorical representations of affect, particularly variations of Ekman's six basic emotions. Behavior and attitude, although rare, are also represented sometimes. Emotions like anger, happiness, and sadness receive the most attention, while the recognition of the neutral state as an emotional state remains controversial. Dimensional affect representation schemes are less common, possibly due to the difficulty in estimating valence solely from audio input. Researchers often combine multiple categorical schemes to accommodate different datasets used in SER systems, aligning the popularity of the schemes with the corresponding datasets. However, issues such as a lack of explanation for chosen categories, interchangeable use of terminology, and a weak psychological foundation for category selection pose challenges in achieving a comprehensive understanding of affect representation in SER research.

In human-human interactions, the majority of information is conveyed through body language, specifically facial expressions. Consequently, researchers have been interested in improving human-computer interactions through developing systems with automatic understanding of body language and facial expressions.  This technology is especially useful due to its broad range of applications in fields such as healthcare, education, and safety & security. Vision-based automatic affect recognition (AAR) systems aim to predict a subject’s affective state based on visual input such as image or video. These systems analyze and classify subjects’ facial expressions and body language using affect representation schemes (ARS), most often classified as either categorical or dimensional. This paper explores the current state of ARS used in vision-based AAR through a systematic literature review following PRISMA guidelines. We selected 53 papers from WebOfScience according to our eligibility criteria which included computer science papers written in English proposing a vision-based AAR system targeting single subjects, and excluded studies dealing exclusively with micro-expressions or group affect recognition. Additionally, given the time limitation imposed on this research we excluded papers that were not readily accessible with our TU Delft license, used multimodal input, or did not use a dataset included in our  predefined list. For this exploration we specifically look at the schemes used, the popularity and trends of usage, motivations, and psychological basis. From the 53 reviewed papers, all of the papers target utilitarian emotions using at least one discrete ARS. The most commonly used schemes classify affective states into happiness, sadness, fear, anger, surprise, and disgust. While the majority of papers are lacking in providing explicit reasoning for their choices, most ARS are based grounded in psychological theories. Our results show an established norm within this area of research. However, they also evidence a lack of displayed critical thought in the selection of schemes. This oversight limits potential for future AAR research.

This research delves into the exploration of translation methods between affect representation schemes within the domain of text content analysis. We assess their performance on various affect analysis tasks while concurrently developing a robust evaluation framework. Furthermore, we collect annotated datasets and take into account crucial contextual and individual factors. Ultimately, our goal is to contribute to the advancement of powerful and sophisticated tools for affect analysis. We believe a successful automated translation will aid in achieving a more comprehensive and rounded understanding of affect and further research in different fields, such as psychology and sociology.

The objective of this report is to establish and present a machine learning model that effectively translates affect representation from emotional attributes such as arousal (passive versus active) and valence (negative versus positive) to dominance (weak versus strong). In the pursuit of this goal, various research questions are addressed. The paper outlines the process of dataset selection, ensuring appropriateness for the problem at hand. Subsequently, a comprehensive investigation into suitable evaluation methods for the developed model is conducted, providing well-reasoned justifications for the chosen approach. An additional research question focuses on assessing different machine learning approaches to determine the optimal performer. The motivation behind this translation lies in the recognition of the interdependence between these affect attributes, supported by both theoretical underpinnings and practical evidence. This contrasts with previous studies that have treated these dimensions as independent descriptors for representing emotions.

Living in a world where every single electronic device is online and interconnected, privacy is a growing concern. Finding the threshold where audio is unintelligible to transcription software is crucial when everything that we say can be recorded. Even if Automated Speech Recognition (ASR) is used in tools, such as Siri or Alexa, designed to ease daily tasks, it could also be used in malicious manners. ASR technology has not been around for too long and like any other new piece of technology, it still has many aspects that have not been looked into and are unknown to the public. This research paper addresses this knowledge gap by examining how sample frequency reduction affects word detection using current well-known transcription software technology such as Google’s speech recognition software and Kaldi’s toolkit. The behavior and performance of these two software pieces have been analyzed for different sample frequencies in the range from 300Hz to 44,1kHz.