Odour-cued memory retrieval has emerged as a potential intervention for people with dementia due to the unique properties of odour-evoked memories. The design and application of technologies that integrate this strategy require knowing which odours are meaningful to a given person, motivating the systematic modelling of odour-event associations across the lifespan. In this paper, we introduce a computational pipeline that uses a Large Language Model (LLM) to simulate diverse persona profiles and model these associations at scale. We constructed persona-event pairs by operationalising age, gender, nationality, origin environment, and personal values. Then we prompted the LLM to generate an odour description and its most likely source for each pair. A bottom-up open card sorting taxonomy is then applied to cluster the odour sources into 13 odour types, forming an exploratory olfactory memory map, while events are also clustered similarly into 33 event types. Empirical analyses confirm the significance of systematic odour-event associations being encoded in our LLM-generated sample. While interaction effects for lifespan were not significant, the total effects for events, lifespan and cultural indicators were significant. The interaction effects of our cultural indicators were also significant for our sample. This study is a step towards using LLMs as a computational proxy in the field of LLMs for personalised assistive technologies to be adaptive and culturally inclusive in order to aid patients with dementia. ... Read more

Visitors of public security facilities (i.e. detention centres, courthouses, penitentiaries and customs) often experience anxiousness and low mood during their time in the waiting room, while staff have limited capacity to attend to visitors. To address this, we explore the design of a social robot that engages visitors through socially appropriate interactions intended to support a positive mood, while simultaneously providing staff with remote situation awareness. Simultaneously, social robots have been deployed as reception agents and mood-intervention tools in public settings (e.g., healthcare[1] and retail[2]). Usually, social robots in public buildings handle welcoming and giving information; this thesis will test what happens when a reception robot also provides visitors a mood-intervention activity. A social robot could therefore provide a friendlier atmosphere in the emotionally fraught visitor area of public security constrained facilities. Here we identify three complementary robot roles – receptionist, storyteller, and playmate – combined with a monitoring function for staff. Receptionist and storyteller roles were lab-evaluated using a local LLM powered social robot as a mood-intervention agent in two iterations with a Before-After Control-Impact (BACI) study. In the lab, the design was implemented with a Softbank Pepper-robot, a pre-post study indicated that interaction effectiveness depends on turn-taking level: high turn-taking activities produce larger immediate mood gains than low turn-taking storytelling. The playmate role was field-tested in a security-constrained public service waiting room. Beyond visitor surveys, semi-structured interviews explored staff acceptance. In the real-world evaluation, the design was implemented with an Enchanted Tools Miroki robot. The results indicate that visitors particularly value the playmate role, the guessing game agent was well accepted: 89% of visitors self-reported that the robot improved their visit(t(9) = 4.0, p < .005). Staff interviews underline acceptance rests on sociotechnical framing, perceived operational usefulness and minimizing of operational overhead. The results advance understanding of social robots in high-consequence domains and other affectively complex (anxiety-laden) public places, suggesting that rapid, reciprocal interaction designs improve mood-intervention outcomes for security constrained facility visitors. Overall, the study findings inform the iterative design of social robots for sensitive, high-security public environments. Our findings suggest further elaborating the waiting room robot with a (1) time-aware conversation system to offer activities suited to visitors’ time-frames and (2) multilingual functionality for visitors without Dutch proficiency. Additionally, the other areas of the security constrained facility provide possibilities where robots may help reduce staff burden; (3) as a multilingual interpreter and as a (4) supporting buddy to reflect on accidents during visit to the facility.
... Read more

As robots increasingly transition from automated tools to collaborative teammates, trust becomes a central requirement for effective human–robot collaboration. While prior research has examined trust in human–robot interaction, little is known about how trust dynamically unfolds across its full trajectory of formation, violation, and repair, particularly in comparison to human–human collaboration and within physically co-present settings. This thesis investigates how a teammate’s identity, human versus robot, shapes the development of interpersonal trust over time.

A controlled laboratory study was conducted in which participants collaborated with either a human confederate or an anthropomorphic robot teammate on a cooperative building task requiring high interdependence. Trust was measured across three phases: initial collaboration (trust formation), a competence-based mistake (trust violation), and a subsequent repair attempt involving an apology, explanation, and promise (trust recovery). Trust was measured using trust questionnaires capturing trusting beliefs and trusting intentions. Data were analyzed using a Bayesian multilevel modeling approach to account for repeated measures and individual differences.

The results show that participants initially reported lower trust toward the robot than toward the human teammate. Contrary to expectations based on the perfect automation schema, trust declined more sharply following a mistake by the human than by the robot. During the recovery phase, trust rebounded in both conditions. Trust toward the robot recovered to its initial level, while trust toward the human did not fully return to baseline.

Analyses across trust dimensions further revealed that benevolence perceptions toward the robot improved over time, narrowing the initial gap between human and robot teammates. Competence perceptions showed similar violation and recovery patterns across conditions. In contrast, trusting intentions showed a more uneven pattern: although willingness to rely on the robot seemingly returned to its own baseline during recovery, the human–robot difference widened again at \(t_3\), suggesting that reliance remained more sensitive to teammate identity even as other trust dimensions converged.

Overall, this study demonstrates that trust toward human and robot teammates follows similar formation, violation, and recovery phases, but differs in how changes are anchored to initial expectations and distributed across trust dimensions. Specifically, participants began with lower trust in the robot, yet a human teammate’s mistake produced a sharper drop and less complete return to baseline than a comparable robot mistake. While trust toward the robot increased relative to its own baseline, particularly through benevolence, willingness to rely remained more differentiated by teammate identity. These findings show that aggregated trust scores can mask dimension-specific dynamics and that recovery in trust beliefs does not necessarily translate into equivalent recovery in trusting intentions. Practically, this suggests that designing for effective human–robot teamwork requires addressing not only how robots regain positive evaluations after errors, but also how to support users’ willingness to rely on them in interdependent tasks. ... Read more

Human-agent teaming in high-stakes domains is already contributing positively to society, yet these AI agents are often still tools directly controlled by humans. Becoming teammates requires agents to be more autonomous and interdependent, two factors that determine what humans need to know about agents. Agent transparency and explanations can provide this necessary knowledge for effective and responsible collaboration. However, we lack an understanding of what agents should disclose and clarify across interdependencies and autonomy levels. Accordingly, this thesis examines how to design transparent and explainable agents that foster effective and responsible human-agent teaming.

We first develop a conceptual framework that distinguishes agent transparency (disclosing information) from explainability (clarifying that information) and relates these concepts to interpretability and understandability, resolving common ambiguities. Using simulation environments, we then demonstrate that interdependence influences how transparency and explanations impact human-agent teaming processes, underscoring its importance in studies on transparent and explainable agents. Next, we examine the trust calibration process across interdependencies. We find first evidence that interdependence relationships influence trust calibration in human-agent teams, suggesting that engaging in joint actions facilitates more accurate trust calibration.

To support responsible human-agent teaming, we develop an evaluation method for meaningful human control based on expert knowledge, operationalizing traceability through objective and subjective indicators and eliciting reasons underlying outcomes. We apply this method to study agent autonomy and explanations in morally sensitive situations. The findings suggest that people prefer more involvement over greater agent autonomy and that they take on greater moral responsibility when agents explain potential consequences. These insights are crucial for designing agents that enhance human moral awareness and human-agent teaming in morally sensitive situations.

Translating these insights to practice, we design TEAMS (Transparent and Explainable Autonomy for Mapping and Searching). This human-robot collaboration system for firefighting moves beyond teleoperation by proposing and explaining intermediate navigation destinations while autonomously navigating towards them. This system is grounded in expert firefighting knowledge and can address the challenge of camera-based teleoperation in low-visibility conditions. We highlight the importance of training, iterative and human-centered refinements, and software optimization to further enhance the system.

Finally, we synthesize a research agenda with taxonomies and guidelines, team design patterns, modular testbeds, and study templates to advance the field. Taken together, this thesis offers a path from concept to practice: a conceptual framework, studies in simulation environments, an evaluation method for meaningful human control, and TEAMS in a practically grounded setting, complemented by a research agenda. By doing so, this thesis supports the design of transparent and explainable AI agents that foster effective and responsible human-agent teaming. ... Read more

As machines take on more complex tasks, we move from asking how well they can perform those tasks to asking how well they can collaborate with us. After all, the goal of building technology should be to improve our lives, not make them harder, but that requires mutual understanding, coordination, and trust. This dissertation looks at the role of trust in decision-making within teams of humans and semi-autonomous machines, including AI systems, agents and robots. In particular, we look at the concept of artificial trust, that is when an artificial agent reasons about someone’s trustworthiness.

Trust is central to human decision-making. When we work with others, we constantly judge who is reliable and who is not, and we delegate tasks based on how trustworthy we think our teammates are and what risks those choices pose to us individually and to the team as a whole. When we see someone is not very trustworthy for a task they are expected to perform, and that poses risks to them or us, we can also offer help. The same logic can extend to artificial agents. When humans and intelligent artificial agents work together, artificial agents must not only be trusted by humans but also develop ways of assessing how trustworthy their human partners are for different tasks. In other words, artificial agents can use artificial trust to make decisions. This requires defining, modelling and using trustworthiness for decision-making in human–agent teamwork. We go over all of those steps in this dissertation.

This research argues that human trustworthiness is not only about a few internal traits such as ability, benevolence or integrity. In fact, what counts as trustworthiness can vary depending on the task and team characteristics. For example, if success in a task depends only on being somewhere on time, then punctuality may be the only relevant trait. Furthermore, to perform a task successfully, a person not only needs to be able to do it but also needs to choose to do it. Our research shows that in human–agent collaborative scenarios, task choices can often be explained by contextual cost–benefit reasoning. People consider a task by weighing its potential benefits, such as reward, against its potential costs, such as effort and time. This translates into a person’s willingness to do a task. At the end of the day, it is not enough that someone has the skills to succeed in a certain task, but it is also important that they are willing to do it.

Although it is challenging to infer someone’s willingness for different tasks, both for humans and machines, we can try to find ways around it. For example, asking directly about teammates’ competence and willingness can give machines better information to work with, helping them to make fairer, more transparent and more efficient decisions. One of our studies found that people want artificial teammates, such as robots, to consider their preferences and willingness, but only in non-critical situations. In urgent or high-stakes work, efficiency mattered most. However, over time, recognising willingness may help make collaboration more sustainable and engaging.

This dissertation focusses on developing machines that can complement and even augment human teams, instead of replacing people. For that to happen, we need a solid understanding of how people make decisions, what motivates them, and what they value in teamwork and in their artificial teammates. At the same time, giving machines the power to trust or distrust humans raises ethical risks. Used wrongly, it could harm individuals or undermine their autonomy. These concerns are especially pressing in areas such as defence, where collaborative technologies are already being explored, and can contribute to the escalation of armed conflicts. As such, the goal of this dissertation by building artificial trust is not to maximise efficiency at all costs. Instead, we hope to help design systems that support human well-being, safety, and dignity. This requires combining theoretical and technical advances from different disciplines, such as the social sciences and computer science, and carefully reflecting on the contexts where these systems are deployed.
... Read more

This project investigates the design process of a straightforward, user-friendly interface that enables cooperative storytelling between a person with dementia and a family member, through the mediation of a social robot. Due to ethical constraints, evaluation was conducted only with a medical specialist and HCI students. Low cognitive load, sequential navigation, and multimodal visuals are given top priority in the final prototype. Working without direct user input and striking a balance between functionality and simplicity were two major obstacles. The project contributes to the development of supportive, dementia-friendly human-robot interfaces and provides a basis for future testing with individuals who have dementia. ... Read more

Storytelling has many benefits for people with dementia (PwD), such as improved well-being, confidence and communication. However, there is not much research on robots conducting such activities in dementia care. Thus, the goal of this project was to implement a system that can facilitate a storytelling session between PwD and their family members, keeping them involved through the entire process.

The system uses a Large Language Model (LLM), specifically Gemma 3, to generate responses to user's messages based on carefully written prompts. Other strategies used to facilitate this system are separation of the storytelling phases, turn-taking, story personalisation based on participant's wishes and empathetic responses to participant's responses. To test the system, personas based on PwD and their family members were used.

The results showed that the system successfully facilitates a storytelling session, following the different phases and keeping the participants involved in the process. However, some unexpected behaviour was noticed, for example not switching from one phase to the other on time. These findings demonstrate the potential of LLM-based storytelling robots in dementia care, while also underlining the need for further refinement and testing with real users. ... Read more

This paper presents a unified evaluation framework for assessing multimodal storytelling robots used in dementia care. Dementia increasingly affects the quality of life of older adults, and co-creative storytelling with social robots has shown promise in supporting social engagement and emotional well-being. However, existing evaluations often overlook whether generated content fairly reflects the contributions of people with dementia (PwD). To address this, a framework is proposed that jointly evaluates the accuracy of textual, visual, and audio outputs to the original input and their emotional coherence. The method incorporates alignment metrics (AlignScore and BERTScore) for text, image relevance (VQAScore), and audio emotion analysis (valence-arousal), as well as speaker attribution to ensure equitable representation. Results from experimental sessions show that data biases can be quantitatively identified and correlated with user enjoyment indicators. These findings offer a scalable approach to evaluating storytelling robots, ensuring both therapeutic benefit and respect for user identity in sensitive care contexts. ... Read more

Personas are a particularly useful testing tool for storytelling robots for people with dementia (PwD) because they offer an alternative to direct user involvement, which is often limited by recruitment, privacy, and consent-related challenges. The manual creation of realistic personas is often complex and time-intensive, whereas pretrained large language models (LLMs) offer a promising alternative due to their impressive zero-shot and in-context role-playing performance. This study investigates whether commercially available LLMs can accurately simulate personas of PwD for use in testing of a storytelling robot for PwD. To this end, we developed a custom probabilistic system based on a single prompt chain composed of multiple few-shot and zero-shot prompts, along with an independent storage system for custom memory manipulation. The simulated personas underwent repeated assessments using the Mini-Mental State Examination (MMSE), a standardized assessment for evaluating memory, comprehension, and executive function. Results demonstrated statistical similarity to real scores and indicated that LLM-based personas can closely mirror many of the cognitive profiles characteristic of these conditions: early-stage Alzheimer's personas exhibited marked impairments in recent memory, late-stage Alzheimer's personas showed significant global cognitive impairment, and vascular dementia personas displayed relatively preserved memory but reduced executive functioning. These findings indicate that pretrained LLMs possess the capability to simulate accurate personas of PwD to a significant extent. ... Read more

Recent advancements in pose estimation, activity classification, and explainable artificial intelligence (XAI) have opened new opportunities in sports analytics. However, their combined application within the domain of cricket remains largely unexplored. This paper investigates the integration of XAI methods to interpret black-box models trained to classify cricket shot techniques using pose keypoints extracted from video data. I implement and compare several techniques, including SHAP, Grouped SHAP, Feature Importance, Permutation Importance, LIME, Grad-CAM, and Accumulated Local Effects (ALE) plots based on their ability to generate meaningful and interpretable explanations. Through a structured experimental pipeline involving MediaPipe-based keypoint extraction, random forest (RF), convolutional neural networks (CNNs), and multiple explanation methods, I evaluate which techniques most effectively highlight the body keypoints critical to accurate shot classification. The findings indicate that SHAP outperforms other methods due to its ability to generate both local and global explanations, along with intuitive visualizations. This work contributes to the development of transparent sports AI systems and lays the foundation for future applications of interpretable machine learning in athletic coaching and skill assessment. ... Read more

In this thesis, we focus on developing behaviours for socially interactive agents (SIAs). The context in which the agent is used is a self-regulated learning system for children. We focus on personalising learning objectives and interaction content within an intelligent tutoring system (ITS). We envision a system where children can train diabetes selfmanagement knowledge and skills independent of space and time and in collaboration with the health care professional, legal caretakers, and a SIA. To facilitate long-term interaction with such a system, relevant learning content and appropriate ‘intelligent’ social behaviour of the SIA are necessary. The envisioned system was developed within the Horizon 2020 PAL-project and evaluated in an iterative design process. The main contributions of the research described in this thesis are: insights into the behaviour design for a NAO robot and its virtual avatar, and the formalisation of learning objectives facilitating personalised learning content.
Most studies on SIA behaviour focus on the design of emotional expressions or implement roles (e.g., peer or tutor) that were not validated for perception. We argue that strategical pedagogical interaction style (i.e., style purposefully selected based on knowledge about the user, task and context such as done by teachers in traditional classroom settings) is necessary but not yet sufficiently studied to design meaningful interactions that surpass the initial novelty and fun. Further, we argue that learning content must be relevant to the child’s needs and developmental stage. These two challenges are the subjects of study in the two parts of this thesis.

The main research question addressed in part I is: How to design SIA behaviours that express different pedagogical styles and what is the effect on learning outcomes? We answer this question in the four included chapters.
In a systematic review we focus on non-verbal expressions by parameter-based manipulations of bodily shape and motion of humanoid robots and virtual agents, and how these manipulations are perceived by humans. We present a comprehensive review of peer-reviewed published articles and analyse and summarise the available work. Research in this field is multidisciplinary and shows a large variety in concept definitions, behavioural manipulations and evaluation methodologies. We developed the TAXMOD taxonomy as a starting point to develop a shared understanding and interpretation of research
objectives and outcomes, and to formulate a road-map. We applied TAXMOD to position and compare research, and to explicate progress in this area. We found structural support for the fact that some social signals can be displayed by behaviour manipulation in the form of posture- or motion modulation or designed key expressions (fixed behaviours with a specific target expression). Key findings include: 1) the expression of personality traits using virtual or robot bodies is limited to the trait extraversion; 2) the expression of social dimensions such as warmth, competence and dominance is possible, but only when using the whole body, and more research is needed to disentangle individual effects on friendliness, competence and dominance; 3) the expression
of emotion is restricted to generic positive versus negative signals; and 4) context seems important for users for the correct interpretation of the expressive behaviour.
In a first perception study we evaluate an educational robot displaying non-verbal behaviours expressing high or low warmth and competence with children at primary schools and a camp. We show that style expression by a humanoid robot is possible. Bodily posture, hand gestures and paralinguistic cues were manipulated to evoke an expression of a specific level of warmth and competence. The competence dimension in our model was successful. Warmth manipulations were perceived as intended only in combination with high-competence. Moreover, context influenced children’s perceptions:
at school the robot was perceived warmer and more competent than at camp.
In a second perception study we evaluate an educational robot displaying non-verbal behaviours expressing high or low dominance. We modulate bodily posture and movement, specifically by manipulating body expansiveness. We show the validity of body expansiveness modulation for dominance expression in both postures and gestures and showthat with a limited set of parameterswe can moderate dominance expression. Specific postures and gestures have a natural tendency towards being perceived as more or less dominant. Further, the manipulation effect is consistent for a variety of behaviours except a sitting pose. This study provides evidence that body expansiveness is an important factor for dominance expression and that this effect is independent of specific behaviours and view angle.
We study the effect of stylised behaviours on children’s learning approach and learning gain by having a NAO robot guide children while performing an inquiry-based science learning task where children roll rollers down a slope to discover laws ofmovement, friction and gravity. Robot style is implemented as variations in verbal strategy and nonverbal style expression, resulting in an expert or facilitator interaction style. No effect of robot interaction style on children’s learning approach or gain is reported. Based on only verbal behaviour variations children perceive the explaining robot (either the expert
style or explaining verbal strategy with neutral non-verbal behaviour) as more competent than the robot giving evidence descriptions (either the facilitator style or evidence descriptions verbal strategy with neutral non-verbal behaviour). These perception differences did not impact the learning approach or gain in the present study. We did not find perception differences based on the variation of non-verbal behaviour. We did find that the presence of a robot giving feedback on children during rolling trials did cause children to play longer and do more informative experiments compared to no feedback. However, this difference in learning approach did not impact learning gain.

The main research question addressed in part II is: How to personalise learning content based on personal learning objectives?
First, we look into how learning goals are formulated in pedagogy and ontologies for education: effective learning goals must attune to the appropriate and desired difficulty level. A way to structure this is Bloom’s taxonomy. A learning goal must also have attributes presenting relations between and descriptions of goals. Then, we model educational objectives (i.e., achievements, learning goals and accompanying tasks) in an ontology. The upper ontology structures the classes and relations and defines domain independent constructs (i.e., level and topic). The domain model specifies diabetes self-management training objectives for young children based on current checklists and expert input. The resulting knowledge base was considered relevant to, and covering,
the diabetes domain to a considerable extent. From this we conclude that our upper model adequately supports the formalisation of implicit knowledge of health care professionals on diabetes self-management training. A field study with children with type 1 diabetes in the Netherlands and Italy showed that an SIA-ITS offering tasks based on our model to support basic needs for autonomy, competence, and relatedness of children with diabetes. For the formalisation of domain specific learning goals, achievements, tasks and materials in the knowledge structure we recommend the following design
guidelines: work in a multidisciplinary team (to define an inventory of important learning goals and define learning activities include domain- and pedagogic experts next to knowledge engineers); formulate achievements from logical learning units (e.g., daily challenges) that require a subset of the knowledge and skills encapsulated in the goals to improve relevance; formulate achievements and goals from the perspective of the child to facilitate ownership and increase experienced relevance; and, define user characteristics relevant to goal and/or task selection. For the integration of the knowledge structure in a multi-modal intelligent tutoring system we recommend the following design guidelines: provide instruction and explanation to the child on how achievements, goals and tasks are selected and can be attained (i.e., that progress on a goal is gained by task completion, and benefits earned by this); embed the objectives in the ITS application to make them easily accessible to the (child) user and integrate them in other system functionality such as feedback on progress provided by a SIA; and offer sufficient learning content such as games and quizzes to maintain interest and engagement.
We developed an authoring tool with a tree-based interface adapted from game design for collaborative personal goal setting and monitoring that implements the ontology of diabetes self-management education, and we co-evaluated this interface with health care professionals. We propose the following design guidelines for an authoring tool: provide clear, visual feedback on goal structure, and active state and progress; consistently use a different representation (e.g., shape) for different concepts of the model (e.g., goal and achievement); cover the full domain and different skill levels with the finite
set of goals; and, support assessment of current abilities next to goal setting, progress monitoring and goal attainment registration.
We developed an mHealth dashboard as interface for personal goal and task selection and monitoring, and co-evaluated this interface with children with diabetes. The interface implements our ontology of diabetes self-management education. The following design elements were understandable for all children: colouring indicating status, and navigation between layers of information. Children experienced difficulties interpreting the meaning conveyed in iconic presentations, understanding of the layered information, and navigation. Based on reported usability issues, we present guidelines for
the design of a dashboard for children: provide descriptive labels next to visual elements because children lack experience using apps and thus understanding of icons and such; connect elements accordingly by placing them in close proximity and in boxes with appropriate labels; ease navigation between layers when hiding detailed information to avoid cognitive overload; and avoid cluttering elements such as navigation bars.

The work in this thesis shows that robots can express different pedagogical styles perceivable by young children. Dominance expression is mainly dependent on body expansiveness. Warmth and competence expression rely on a complex set of behaviour modulations. However, current style variations are too subtle to impact learning approach and gain. With respect to content personalisation, we show that a structure for and selection of learning objectives provide both a personalised learning path as well as personalised content.
Overall we conclude that to impact learning approach and gain not only SIA behaviour must be modulated, it must be noticed by the learner as well. Learning objectives and content should be formalised within a structure and a user-friendly interface is needed to select objectives and tasks with accompanying content, and monitor progress. The success of an SIA-ITS depends on the amount of available content and social interaction
capabilities of the SIA. ... Read more

This thesis investigates how Reinforcement Learning (RL) can increase support effectiveness in virtual coach-based smoking cessation interventions. Such interventions have shown promise in helping people change behaviors such as smoking. However, personalizing the support they provide by accounting for people's current and future states might further increase their effectiveness. States thereby refer to people's relatively stable conditions at certain moments in time, capturing aspects such as motivation, knowledge, or the presence of personal reminders. After deriving general user needs for the support provided by a virtual coach-based smoking cessation intervention from a study with 671 daily smokers, we thus used RL to adapt the support to people's current and future states. Specifically, using data collected from three crowdsourcing studies with each more than 500 participants, we assessed the effectiveness of different RL model components in adapting 1) how people are persuaded, 2) what they are asked to do, and 3) who they are supported by. Our findings suggest that considering current and future states increases the effort smokers spend on smoking cessation activities and helps them build quitting-related competencies over time. Given that model components were derived from behavior change theories, this shows the potential of using psychology-informed RL to create smoking cessation support that is effective in the long run. ... Read more

Intelligent machines (in the form of physically embodied robots or virtual agents) are increasingly able to perform tasks in collaboration with humans. However, learning to become a good team takes time, especially when dynamic tasks require the team to constantly adapt to new situations. Over time, both human and machine need to not only learn how to execute the task, but also how their team partner behaves in the task, as well as how to improve their collaboration over time by attuning their behavior to each other. Existing research on human-machine collaboration often does not sufficiently address adaptation and learning. Work that does study adaptation and learning tends to focus on either machine learning and adaptation or human learning and adaptation, thereby not addressing the interaction of these learning processes that would be present in a co-learning situation... ... Read more

This project aims to harness the potential of music-making and robotic interaction to enhance creative expression and cognitive function among individuals with cognitive impairment and dementia. With the aging population, there is a growing demand for innovative interventions that support cognitive health and active engagement. Music therapy has demonstrated effectiveness in stimulating cognitive function and emotional expression in individuals with dementia. Creative expression through music serves as a unique outlet, fostering cognitive functions and emotional well-being. This study explores the synergy between music therapy and Socially Assistive Robots (SARs) to develop a more immersive therapeutic intervention. It represents an exploratory investigation into an end-to-end robotic intervention, proposing various interaction elements and examining their functionalities. Each element is designed to foster engagement, enhance perceptions of collaboration, and promote feelings of creativity.
In this thesis, we propose an end-to-end interactive music-making experience designed for use with the Pepper robot, an SAR. The system features a user-friendly interface with eight color-coded boxes, each corresponding to a musical note. Users simply tap the boxes to create melodies. The Pepper robot acts as a guide, assisting users in interacting with the interface. It additionally implements an engagement tracking system by monitoring user interaction through the screen taps on the interface and provides real-time feedback and encouragement. If a period of inactivity is detected, Pepper gently nudges the user to re-engage. Furthermore, the robot functions as a collaborative musical partner, providing rhythmic accompaniment if the user desires. The system also records user-created music and provides playback functionality, allowing users to revisit their compositions.
Methodologically, the study involves an end-to-end system comprising an intelligent music-making interface and an interactive robot providing real-time feedback and rhythmic accompaniment. Insights from the exploratory study highlight the benefits of real-time feedback in enhancing engagement, particularly among participants with musical backgrounds. However, rhythmic accompaniment shows mixed results in fostering collaboration, indicating a preference among participants for emotional connection in collaborative settings. Since this is an exploratory study, the empirical study focuses on healthy older adults, a population with an increased risk of cognitive decline. This is because individuals with dementia are a vulnerable group. Music interventions have shown promise in improving cognitive function and engagement in individuals with dementia. Therefore, this study informs the design of future interventions for people with early-stage dementia.\\
Key findings underscore the potential of real-time feedback and interaction in promoting engagement in the activity. The intelligent music interface also shows potential to support creative exploration, albeit with improvements needed for advanced musical participants. Participants appreciate the playback feature, enhancing their sense of creative ownership and motivation. Despite promising outcomes, the study acknowledges limitations in sample size and participant demographics, primarily recruiting from music-engaged older adults rather than the target demographic of individuals with cognitive impairments.
Future research directions include expanding participant diversity, refining robot interaction capabilities, and addressing technical challenges to improve system usability and accessibility. Integrating findings from ongoing research on music and memory could further enhance personalized interventions. Ultimately, this study lays the groundwork for future developments in robotic interventions that promote well-being through music therapy for individuals with cognitive impairments. ... Read more

This research explores the integration of episodic memory and visual short-term memory in social robots to enhance engagement and interaction with older adults, particularly those with dementia, through art activities such as painting. Dementia significantly impacts millions globally, being one of the leading causes of death worldwide. Furthermore, it creates a substantial burden on caregivers and healthcare systems. This research aims to leverage social robots to provide meaningful engagement, companionship, and emotional support to individuals with dementia.

The study involved developing a painting application using Android Studio and a conversational agent using generative AI. The AI model was enhanced with episodic memory and visual short-term memory, enabling it to reference past interactions through episodic memory and analyze painting progress through recent visual inputs with short-term memory. This qualitative study included 8 participants, and post-session interviews were conducted to gain deeper insights into the participants' experiences with the system.

The study suggests that memory-enhanced social robots can improve engagement and interaction quality among older adults interested in painting, but have limited impact on those uninterested in the activity. Mixed responses were observed regarding the robot's role as a companion, with some participants feeling a sense of company while others did not. Overall, participants did not experience frustration due to the robot's presence.

Challenges include response time and real-time awareness, highlighting the need for a more intelligent system. Given the small sample size of eight participants, future research should involve larger groups for more comprehensive and reliable data. ... Read more

Recent studies have shown that gesture annotation schemes should account for the multidimensional nature of gestures and define their meaning in terms of referentiality and pragmatic meaning. However, accurately annotating gesture meaning in densely crowded social settings using such a coding scheme remains to be accomplished. This study uses the MultiModal MultiDimensional (M3D) labelling scheme and the EUDICO Linguistic Annotator (ELAN) tool to annotate video data from the Conference Living Lab (ConfLab) dataset. The ConfLab dataset contains 8 video recordings of standing conversations at a conference, captured from an overhead perspective, and low-frequency audio recordings of the conversations. A total of 1119 clips of individual gesture instances are generated. This data is then fed into a VideoMAE model pre-trained on the UCF101 dataset. The model achieves an overall accuracy score of 49% on the test set but shows a significant bias towards one class due to the imbalanced dataset. Due to the small size of the dataset and the similarities between gestures with different meanings, the model cannot identify different gesture types. The results demonstrate that high-frequency audio or transcripts of the conversations are vital to avoid strong and potentially incorrect assumptions when annotating gesture meaning. Further investigation is required into the annotation and classification of pragmatic meanings and Machine Learning solutions for multi-class, multi-label video classification problems. ... Read more

Cognitive Impairment, commonly termed as Dementia, affects a large number of older adults. People with dementia (PwD) experience cognitive decline that impacts their ability to perform daily activities and maintain social connections. The number of PwD is expected to rise, and unfortunately, there is still no cure. A number of non drug therapies such as music and art therapy are used to stimulate neurons and slow down the progress of the disease. When symptoms become severe, full-time care may be needed, requiring a move to a nursing home for formal care. However, studies show that interaction with peers is limited in a care home, leading to loneliness and reduced quality of life. Caregivers organize group activities like singing and baking to alleviate these feelings, but their busy schedule can limit the frequency of these exercises. This begs the question: Can a robot support caregivers in the provision of group activities in a care home? This would allow caregivers more time to attend to the personal needs of the residents. It not only has the potential to reduce workload of caregivers but also creates opportunities for residents to interact with each other.

In this thesis, we propose the design of a robot which moderates a sing-along activity for a group of people living in a nursing home. An activity in the realm of music therapy is chosen since it is one of the non-drug therapies which has shown to have numerous benefits like improving memory recall, eliciting emotions and aiding relaxation. The robot session is curated to give opportunities for interaction with group members and with the robot too. While the song is playing and participants are singing along with the music, the robot performs engagement detection, and encourages those who do not seem involved.

Multiple ways are proposed to detect engagement in HRI, such as emotion recognition and estimation of head pose. These approaches have limitations in the scope of this thesis, particularly because these methods are not designed for groups of older adults residing in a care home. Given these limitations and our focus on the target group, we propose a novel engagement detection methodology, which leverages the presence of a robot and assigns an active role to it. Instead of relying on conventional passive engagement detection methods, this approach uses an interactive probing technique. The robot prompts users to perform certain actions, and whether or not they respond to it is monitored using pose estimation. This is termed robot engagement. This approach is combined with activity engagement, which indicates whether the participant is singing or not. With this hybrid technique, we aim to increase the efficacy of engagement detection. If a participant is detected as disengaged, either in the activity or in the robot, we provide personalized encouragement and motivation by addressing participants by name. Along with this, the robot also compliments those who actively participate, motivating them further... ... Read more

With the advancement of technology and the integration of Large Language Models, interactive robots are entrusted with more significant tasks that contribute to human well-being. This transition is particularly significant in educational settings, where social robots assume diverse roles such as teacher, tutor, and instructional tool, aiming to enhance learning experiences. Engagement emerges as a central theme in Human-Robot Interaction studies, especially within pedagogical contexts, where it has a significant impact on learning outcomes. However, optimizing engagement rather than maximizing it may yield superior learning results. This thesis investigates the correlation between the employment of engagement strategies and learning outcomes in prompt-driven discussions with robots, focusing on facilitating self-reflection and understanding personal values among children. Due to limitations with the target population, a total of 55 university students conversed with the robot in two different sessions, where they discussed different situations, how they would react in each, and the reasoning behind their behavior, using the Schwartz values as a basis for the choices. The participants were divided into two conditions, to examine the effect of techniques such as motivational interviewing, cues to images, feedback, and the use of a memory model on the quality of their arguments and their ability to recollect the interaction. Results suggest that the strategies lead to significantly more reflective arguments for the first session, but they equalize in the second interaction. No significant effect was found between condition and participants' identifying their value profile which led to an examination of the assumptions made by the robot. Participants who agreed with the assumptions were more likely to identify their values correctly, but, generally, the value model was unable to fully capture the intricacies of human motivation. Participants found the robot equally likable, a possible result of the novelty effect. On recollection quality, the data shows that the effect of the session is too strong to allow room for the condition. Overall, this study contributes valuable insights into the influence of engagement strategies on learning outcomes in educational interactions, offering guidance for designing more effective and engaging interactions in the future. ... Read more

Several useful technologies to help people with dementia already exist. Robotic pets, for example, have been shown to alleviate negative symptoms associated with dementia. Most robotic pets, however, only have pet-like functionality. This is useful on its own, but adding additional care functionality would be helpful. A different useful tool for people with dementia is reminder systems. These systems can help them remember to do important daily tasks, like eating meals and taking medicine. These systems, however, usually use a phone as a medium. The user might leave their phone in another room, and if they do not take it to the location where the task is to be completed, they cannot check what the reminder was for.

This thesis aims to combine these two systems, and presents the design of an integrated robot reminder system for people with dementia who live at home. It aims to address the problem of people with dementia forgetting important things, improve their quality of life, and reduce the stress experienced by their informal caregivers. The system consists of an animal-like robotic pet, which delivers reminders for important activities of daily living directly to the user with dementia. It guides them to a location in the house, where a screen tells them what the reminder is for. Through sensors placed in the house, the system can automatically detect whether some reminders have been completed, and can use contextual clues to send reminders at the best time. The research question for this thesis is “How can an integrated robotic reminder system effectively support people with dementia living at home in completing daily tasks while alleviating stress of their informal caregiver?”

Three research topics are explored in the thesis. The first topic is that of value-sensitive design. The two main user groups, people with dementia and caregivers, both have values they want to see reflected in the system. Sometimes, these values may clash. The conflict between these values is explored in this thesis. Secondly, the interaction between the robot and the person in the context of a reminder system is researched. It should be clear to a user that the robot is trying to convey a reminder. The third topic relates to the software architecture and software engineering requirements. Private data should be stored securely, and the requirements should be written to make future development easy. Recommendations are made regarding the ideal setup of the system, ensuring proper security and usefulness. These technical choices were validated and improved through two interviews with experts in the field of privacy.

The system was designed through an iterative process. In order to improve the system and validate its acceptability, two design workshops were held, one with professional caregivers and one with people with dementia living in a care home. We showed them a high-fidelity robot prototype, and showed the caregivers a user interface prototype for a caregiver app. These workshops showed that the combination of the robot and the reminder functionality was appealing, though not everyone was as interested. ... Read more

The promotion of desirable behaviours, such as socially appropriate or health-promoting actions, can be bolstered through a deeper understanding and awareness of the values that underpin the associated behavioural choices. Various implementations for promoting behaviour change based on goals already exist in Human-Robot Interaction, but, since values are the building blocks for our behaviour an agent can use values for behaviour change. By reflecting on value-related choices in a conversation, a conversational agent can assist in identifying the values at stake. This is particularly significant as these agents serve as accessible and non-judgemental platforms for discussing various concerns in a private setting. Such reflective conversations are time-consuming and may span multiple sessions, necessitating some form of memory (e.g., to reflect on earlier statements and compare the choices made for different situations).

While various robotic agents have been developed to provide behavioural support (e.g., for human health), the absence of a comprehensive memory structure and dialogue strategies capable of fostering personalised, reflective conversations based on the appreciation of certain values and actions in various scenarios through contextualised values remains a challenge. To address this, this study introduces the Contextualised Value Model – a dynamic memory model designed to facilitate value-based reflection and support personalised interactions between humans and robotic agents.

To realise this robotic memory, a conversational agent was designed that could elicit values from participants by discussing various scenarios that happen in daily life and reflecting on said values using perspective-taking and other dialogue strategies.

The evaluation of the Contextualised Value Model focused on three primary aspects: the model's accuracy, the influence on likeability and intelligence, and the effect on participants' value awareness. The model was evaluated during a between-subjects experiment (N=54), consisting of two conditions, one where the robot was able to update and use the Contextualised Value Model, and another one where the Contextualised Value Model was random throughout the conversation.

The outcome measures indicated that the integration of the memory model in conversations led to a personalised and relevant conversation, highlighting the potential of the Contextualised Value Model in enhancing conversation personalisation. Although participants' value awareness and perception of the robot's likeability and intelligence did not significantly differ based on the memory model, the study emphasised the need for extended observation to thoroughly evaluate long-term impacts.

Overall, the Contextualised Value Model presents a promising framework for enhancing personalised interactions in various real-world applications, emphasising the need for further research in this area. The ePartner4all project could be further developed to complement the efforts of primary school teachers and parents in supporting children's self-learning of socially, mentally, and physically desirable behaviours. ... Read more