A Visually Grounded Speech model is a neural model which is trained to embed image caption pairs closely together in a common embedding space. As a result, such a model can retrieve semantically related images given a speech caption and vice versa. The purpose of this research is to investigate whether and how a Visually Grounded Speech model can recognise individual words. Literature on Word Recognition in hu- mans, Automatic Speech Recognition and Visually Grounded Speech models was evaluated. Techniques used to analyse human speech processing, such as gating and priming, were taken as inspiration for the design of the experiments used in this thesis. Multiple aspects of words recognition were investigated through three experiments. Firstly, it was investigated whether the model can recognise individual words. Secondly, it was investigated whether the model can recognise words from a partial sequence of its phonemes. Thirdly, it was investigated how word recognition is affected by contextual information. The experiments show that the model is able to recognise words while not being supervised for that task, and that factors such as word frequency, the length of a word and the speaking rate affect word recognition. Furthermore, the experiments reveal that words can be recognised from a partial input of a word’s phoneme sequence as well, and that recognition is negatively influenced by word competition from the word initial cohort. Furthermore, the word recognition in context experiment reveals that contextual information can enhance the recognition of words which are recognised less well.

The promotion of desirable behaviours, such as socially appropriate or health-promoting actions, can be bolstered through a deeper understanding and awareness of the values that underpin the associated behavioural choices. Various implementations for promoting behaviour change based on goals already exist in Human-Robot Interaction, but, since values are the building blocks for our behaviour an agent can use values for behaviour change. By reflecting on value-related choices in a conversation, a conversational agent can assist in identifying the values at stake. This is particularly significant as these agents serve as accessible and non-judgemental platforms for discussing various concerns in a private setting. Such reflective conversations are time-consuming and may span multiple sessions, necessitating some form of memory (e.g., to reflect on earlier statements and compare the choices made for different situations). While various robotic agents have been developed to provide behavioural support (e.g., for human health), the absence of a comprehensive memory structure and dialogue strategies capable of fostering personalised, reflective conversations based on the appreciation of certain values and actions in various scenarios through contextualised values remains a challenge. To address this, this study introduces the Contextualised Value Model – a dynamic memory model designed to facilitate value-based reflection and support personalised interactions between humans and robotic agents. To realise this robotic memory, a conversational agent was designed that could elicit values from participants by discussing various scenarios that happen in daily life and reflecting on said values using perspective-taking and other dialogue strategies. The evaluation of the Contextualised Value Model focused on three primary aspects: the model's accuracy, the influence on likeability and intelligence, and the effect on participants' value awareness. The model was evaluated during a between-subjects experiment (N=54), consisting of two conditions, one where the robot was able to update and use the Contextualised Value Model, and another one where the Contextualised Value Model was random throughout the conversation. The outcome measures indicated that the integration of the memory model in conversations led to a personalised and relevant conversation, highlighting the potential of the Contextualised Value Model in enhancing conversation personalisation. Although participants' value awareness and perception of the robot's likeability and intelligence did not significantly differ based on the memory model, the study emphasised the need for extended observation to thoroughly evaluate long-term impacts. Overall, the Contextualised Value Model presents a promising framework for enhancing personalised interactions in various real-world applications, emphasising the need for further research in this area. The ePartner4all project could be further developed to complement the efforts of primary school teachers and parents in supporting children's self-learning of socially, mentally, and physically desirable behaviours.

The implementation of social robots in the healthcare industry is becoming substantial as a consequence of the scarcity of healthcare professionals, rising costs of healthcare and an increase in the number of vulnerable populations. Social robots will be deployed, in increasing numbers, in assisting health care professionals during the provision of care to patients. While research in Human-Robot-Interaction (HRI) has investigated mechanisms for making decisions taken by artificial agents more ethical, there is limited work done in investigating adaptations to HRI that promotes ethical behaviour on the human-side. Moral dilemmas can appear when decisions have to be made regarding patientcare. Dealing with them can be challenging for healthcare professionals since the impact of decisions affect multiple parties and involves different considerations and value-trade-offs of the involved people. There are different approaches in which healthcare professionals can be confronted with moral dilemmas. Based on decision making principles, a robot might assist the health care professionals in ethical decision making with an appropriate reflection or confrontation of the dilemma. This thesis investigates how a social robot can confront a professional with a moral dilemma and what the associated effects are. Following the Socio-Cognitive Engineering (SCE) method, two alternative proto-patterns are created for dilemma confrontation by a robot in a resuscitation scenario: Verbal and Multi-modal confrontation. In such a scenario, the robot displays protests of distress and affect in order to help the professionals carry out systematic reflection of the moral dilemmas they experience. In an evaluation, it is tested whether these confrontation patterns are being perceived as intended, comparing the two confrontation patterns with a neutral scenario.

Negotiation is not a skill that comes naturally to most people. However, most people could benefit from attaining good negotiation skills. Non-verbal behaviour plays an important role in negotiations. Previous studies have shown a link between mimicry through conversational agents and self-regulation of non-verbal behaviour. The current study aims to raise a person’s self-awareness of their non-verbal behaviour through the medium of a conversational agent in negotiation training. An experiment (N = 64) is conducted where participants performed negotiations with an intermediate feedback round where the participants' non-verbal behaviour was played back on a virtual agent. The agent either mimicked the participants’ non-verbal behaviour or exhibited idle behaviour. It was found that the mimicking agent provided better feedback to the participants. Furthermore, repeated negotiations increase a person's self-awareness. The data indicate that raising a person’s self-awareness in negotiation training by utilising non-verbal mimicry through a conversational agent is feasible.

Human-computer interaction has long been the focus of technological evolution; however, in order for this type of system to reach its peak potential, machines must recognize that humans are constantly influenced by emotions. Text affective content analysis models are one attempt to integrate human psychology into computers, trying to detect the emotion transmitted by written input. There are numerous approaches to implementing such systems, with supervised learning still popular. The challenge of creating textual affective datasets is not in the availability of records, as humanity has reached a peak in data production, especially text, but in ensuring the consistency of the annotations provided by humans when included in the process. This study conducts a systematic literature review focused on providing details of published corpora. The annotation process, as well as any trends in how it evolved, will be examined to obtain dataset particularities. The ultimate intent is to lay the groundwork for an ample study aimed at analyzing the relationship between interrater agreement levels and performance scores of models trained on these datasets. The relevant literature was extracted from 3 search engines: Scopus, IEEE Xplore, and Web of Science, with a focus on manually labeled written records that are not part of multimodal systems, resulting in an analysis of 41 datasets. According to the aggregations, when humans are recruited to perform this task, researchers are more likely to use multiple annotators and calculate the degree of agreement between them to ensure the data's reliability before using it. The conducting researchers are inclined to either train these people before the procedure or tailor the set of labels to potentially increase the uniformity of ratings. As a result, this paper highlights the variety of annotation process characteristics and points towards standardizing this task.

Emotional datasets for automatic affect prediction usually employ raters to annotate emotions or verify the annotations. To ensure the reliability of these raters some use interrater agreement measures, to verify the degree to which annotators agree with each other on what they rate. This systematic review explores what kind of interrater agreement measures are used in emotional speech corpora. The affective states, the affect representation schemes, and the collection method of the datasets as well as the popularity of these measures were investigated. Scopus, IEEE Xplore, Web of Science, and ACM digital library were used to extract papers that describe the creation of datasets. 45 papers were included in the review. The review concludes that the interrater agreement measures used, are highly dependent on the collection method for speech and the affect representation schemes. It was found that there is no standardized way to measure interrater agreement. Datasets that use actors to record emulated emotions mostly use recognition rate as their interrater agreement measures. Datasets that use a dimensional representation scheme often compute the mean agreement of the raters and the standard deviation of that measure to check the interrater agreement. Datasets that do not use actors nor are dimensional use a plethora of different measures such as probabilistic computing of agreement, or majority agreement measures, but a large amount use no measures at all.

This study explores the influence of interrater agreement measures and affect representation schemes in automatic affect prediction systems using physiological signals. These systems often use supervised learning and require unambiguous and objective labeling, a challenge when multiple human annotators are involved, which can affect model performance. The research involved the first part of a two-stage process: systematically reviewing datasets and their characteristics concerning interrater agreement on the affective interpretation of physiological signals. This stage established a reliable foundation for the second step: a future analysis of model performance reported in technical papers utilizing these datasets. The main takeaways were that the number of raters varies significantly over datasets and the complexity introduced by combining affect representation schemes can negatively affect interrater agreement.

This study explores the influence of interrater agreement measures and affect representation schemes in automatic affect prediction systems using physiological signals. These systems often use supervised learning and require unambiguous and objective labeling, a challenge when multiple human annotators are involved, which can affect model performance. The research involved the first part of a two-stage process: systematically reviewing datasets and their characteristics concerning interrater agreement on the affective interpretation of physiological signals. This stage established a reliable foundation for the second step: a future analysis of model performance reported in technical papers utilizing these datasets. The main takeaways were that the number of raters varies significantly over datasets and the complexity introduced by combining affect representation schemes can negatively affect interrater agreement.

Continuous affective self-reports are intrusive and expensive to acquire, forcing researchers to use alternative labels for the construction of their predictive models. The most predominantly used labels in literature are continuous perceived affective labels obtained using external annotators. However an increasing body of research indicates that the relation between expressed emotion and experienced emotion might not be as apparent as previously assumed. Retrospective self-reports provided by participants do capture experienced emotion, but models applied on these labels suffer from the lack of continuous annotations during training. In this work, we aim to answer whether this lack of temporal information can be remedied by using continuous external annotations as proxies for experienced emotion over time. Furthermore, we investigate whether weakly-supervised models can generate accurate continuous annotations to reduce the annotation burden for large datasets. Our results indicate that external annotation sequences bear little significant information for the prediction of self-reports. However, forcing models to reflect changes in external annotations by training models in a multitask fashion improves model performance, suggesting that such temporal supervision helps models to distinguish relevant segments in input data. Besides this, we find that weakly-supervised models can to a certain extent capture changes over time, but in general yield poor results compared to fully-supervised models.

In this work, we present VisuaLayered, the implementation of a combined analysis workflow for pigment identification. VisuaLayered is an integrated, interactive system that focuses on the combined visual analysis of Macro X-Ray Fluorescence (MA-XRF) and Reflectance Imaging Spectroscopy (RIS) data. Analysing paintings for pigment identification is relevant for many applications in the cultural heritage domain, such as conservation and restoration. Domain experts use non-invasive scanning techniques as an initial step in their analysis. Two such techniques are MA-XRF and RIS. They provide hyperspectral data on the elemental and molecular composition of pigments, respectively. Domain experts analyse these two complementary data modalities in order to determine the pigments present in the different paint layers of a painting. However, due to the size and high-dimensionality of these datasets, experts have problems with efficiently analysing the data. In general, they examine the two data modalities separately in the analysis workflow and use their knowledge to unify all the information without additional software support, as there is no integrated system that is designed specifically for the combined analysis of MA-XRF and RIS data. We worked in collaboration with domain experts from the Rijksmuseum, Amsterdam in order to design and implement VisuaLayered based on current domain practices. We use t-SNE for projecting the high-dimensional data into a two dimensional space, which the user can interactively explore in combination with other linked views in order to find connections between the two data modalities. With our system, experts can explore the spatial distribution and the correlation between pixels that have similar molecular and/or elemental compositions. Additionally, for the RIS data, we support endmember identification and analysis based on the pigments' spectral profiles. We tested the efficiency of our system with respect to the designed workflow in a case study evaluation with our collaborator. They successfully used VisuaLayered for the analysis of one painting and found the views combining the two data modalities very useful for better understanding the relation between them. Moreover, they were able to identify new pigments, that they missed when using existing software.

In this thesis the automatic multimodal detection of social and task cohesion in meetings is studied. The presence of social and task cohesion has positive benefits on employee well-being, creativity and productiveness, and can therefore be used to assess meeting quality. Conversational partners imitate each other's body language and speech characteristics to have smoother interactions, to increase liking, and it can cause a coordination of expectations. We hypothesize that social and task cohesion are therefore positively related to the imitation of both body language and speech characteristics. As the group-level alignment of non-verbal speech behaviour has been previously linked to social and task cohesion in meetings, this thesis investigates the relationship between cohesion in meetings and both motion and posture mimicry from accelerometer and video data. Motion mimicry is described using accelerometer features previously used for the detection of friendly and romantic attraction in pairs. We propose a method to convert these features to group-level descriptors of mimicry. The same quantifications of mimicry are also applied to video-based motion quantifiers that have been previously used to detect team cohesion. Appearance is described using HOG descriptors from densely-sampled feature points that are tracked over time. To our knowledge this is the first-time appearance similarity is used to estimate cohesion. We also test if a multimodal mimicry model performs better than a unimodal mimicry model to investigate if the different forms of mimicry contain complementary information. Our group-level movement mimicry features detect social cohesion with average an area under the ROC-curve of 0.64 for social cohesion, and the accelerometer-based movement mimicry features specifically detect task cohesion with an average AUC of 0.63. The multimodal combination of the different features does improve over the unimodal models with an area under the ROC-curve of 0.68 for social cohesion and 0.65 for task cohesion. This shows both that movement mimicry is an indicator of verbal expressions of cohesion, and that measuring mimicry in different modalities can better model cohesion than a unimodal model. Further experiments are recommended for general appearance feature mimicry, specifically in the method used to measure mimicry, to confirm or deny if these are related to cohesion.

Advances in artificial intelligence and machine learning have led to a steep rise in the adoption of AI to augment or support human decision-making across domains. There has been an increasing body of work addressing the benefits of model interpretability and explanations to help end-users or other stakeholders decipher the inner workings of the so-called "black box AI systems". Yet, little is currently understood about the role of modalities through which explanations can be communicated (e.g. text, visualizations, or audio) to inform, augment, and shape human decision-making. This thesis addresses this research gap through the lens of a credibility assessment system. Considering the deluge of information available through various channels, people constantly make decisions while considering the perceived credibility of the information they consume. However, with an increasing information overload, assessing the credibility of the information we encounter is a non-trivial task. To help users in this task, automated credibility assessment systems have been devised as decision support systems in various contexts (e.g. assessing the credibility of news or social media posts). However, for these systems to be effective in supporting users, they need to be trusted and understood. Explanations have been shown to play an essential role in informing users' reliance on decision support systems. This thesis investigates the influence of explanation modalities on an AI-assisted credibility assessment task. A between-subjects experiment (N=375) was performed, spanning six different explanation modalities, to evaluate the role of explanation modality on the accuracy of AI-assisted decision outcomes, the perceived system trust among users, and system usability. The results indicate that explanations play a significant role in shaping users' reliance on the decision support system and, thereby, the accuracy of decisions made. Users were found to perform with higher accuracy while assessing the credibility of statements in the presence of explanations. Users additionally had a significantly harder time agreeing on statement credibility without explanations. With explanations present, text and audio explanations were more effective than graphic explanations. This thesis concludes that combinations of graphical and text and/or audio explanations were significantly effective. Such combinations of modalities led to a higher user performance than using graphical explanations alone.

Robots in classroom settings can help teachers with providing personalised attention to children's health and development. As part of this personalisation, robots should store and use (verbal or multi-modal) information about the children they interact with. One aspect that has been unexplored in existing literature is the detection of memorable moments during these child-robot interaction. Eye-gaze tracking is a low cost and non-invasive method applied widely to gain insight into human's inner processes. This study has found that several state-of-the-art time series machine learning models perform better-than-chance on the detection of memorable moments using gaze tracking. In addition, a shapelet-based transform classifier also performed better-than-chance in distinguishing memories according to $3$ different levels of recall detail. Manual data analysis has identified significantly different gaze behaviour during memorable moments and not memorable moments as well as in the gaze behaviour for different levels of recall detail. The comparison of the results with related literature leads to the hypothesis that memorable moments are likely to be moments of both high levels of engagement and deep thinking. The data analyses also provided insight into children's gaze behaviour for different reasons for remembering a moment. The results show that these reasons, or `internal processes', are distinguishable by gaze patterns and thus provide insight into items or concept that draw the child's attention. This study shows that memorable moments detection for children is a developing and promising field that could potentially provide a lot of insight into children's situated thought processes.

Existing content-based image retrieval models work well for natural photos, but not for images of architectural floor plans. Previous work on floor plan retrieval has focused on graph-based methods, rather than image-based floor plans. Training a CNN-based representation learning framework on segmented floor plan images with standard image augmentations does not result in semantically meaningful retrievals. This work shows that a CNN-based representation learning model can learn features for retrieving floor plans that have similar graphs given the right training signal. Two methods were investigated here: GeomPerturb, a data augmentation that perturbs the underlying geometry of a floor plan, and a weakly supervised method with labels based on the graph edit distance between a pair of floor plans. The results show that while GeomPerturb learns representations that are correlated with the floor plan graph, training with GED labels leads to better retrievals both in terms of the floor plan graph and with respect to room shapes.

In this work, a conversational interaction was designed and implemented to test the effect of references to past events or shared experiences rephrased into motivational phrases within the context of working towards a diet related goal that can assist with type II diabetes over multiple sessions. Prior works that utilized a memory typically did not utilize a memory to refer to the past within conversations and when past events were referred to within a conversation, they were not utilized with motivational rephrasing. Most prior works did not analyze long term interaction or how references to past events should have been utilized. To further research in this area, the following research questions were posed. 1.) Does referring to previous sessions improve goal attainment? 2.) Does referring to previous sessions improve user experience? 3.) Does a variety of references differ from making the same reference? To determine the answers to these questions, interactions were carried out through Zoom in a between subjects experiment with three groups of participants. The results, once analyzed, found that there was a significant difference between interactions that only make references to the same shared experience and interactions that make references to various shared experiences where various shared experiences resulted in a significant increase in the change in motivation from prior to the experiment to during the experiment. This offers the following contributions: 1.) Provide clarity on how shared experiences should be used and not only whether they should be used. 2.) A motivational memory utilizing shared experiences with intrinsic motivational value 3.) Determine the effect of motivational references on goal achievement and user experience. This holds promise for the use of such a memory for goal based efforts and future work can further the domains of application and effectiveness of the motivational memory and shared experiences.

Hate speech detection on social media platforms remains a challenging task. Manual moderation by humans is the most reliable but infeasible, and machine learning models for detecting hate speech are scalable but unreliable as they often perform poorly on unseen data. Therefore, human-AI collaborative systems, in which we combine the strengths of humans' reliability and the scalability of machine learning, offer great potential for detecting hate speech. While methods for task handover in human-AI collaboration exist that consider the costs of incorrect predictions, insufficient attention has been paid to estimating these costs. In this work, we propose a value-sensitive rejector that automatically rejects machine learning predictions when the prediction's confidence is too low by taking into account the users' perception regarding different types of machine learning predictions. We conducted a crowdsourced survey study with 160 participants to evaluate their perception of correct, incorrect and rejected predictions in the context of hate speech detection. We introduce magnitude estimation, an unbounded scale, as the preferred method for measuring user perception of machine predictions. The results show that we can use magnitude estimation reliably for measuring the users' perception. We integrate the user-perceived values into the value-sensitive rejector and apply the rejector to several state-of-the-art hate speech detection models. The results show that the value-sensitive rejector can help us to determine when to accept or reject predictions to achieve optimal model value. Furthermore, the results show that the best model can be different when optimizing model value compared to optimizing more widely used metrics, such as accuracy.