In multi-member human-agent teams the communication and shared mental models within the team are essential for good teamwork and team performance. In some ways the mediating processes are even more important than in human-only team because the artificial agents of today lack many of the innate social behaviours that humans naturally possess. Research into human-agent teams have allowed designers of such teams to anticipate for complex interactions such as trust violation and repair scenarios. In this study a human-agent-agent team undertakes a search- and rescue mission with the human in a leading role, one of the agents free-roaming and the other agent under the human's direct control. Approximately one-third of the way through the mission, the autonomous agents initiated actions independently of human approval, thereby undermining operator trust. As a trust repair strategy the agent employs a promise to do better and a novel authority change by lowering its level of automation and presenting the option of restricting cooperation with the other agent.
We conducted the experiment with thirty participants divided into a two groups with differing trust repair strategies (promise only, promise with the authority change) and measured trust perception at three different time steps.
Results show no significant difference between the two trust repair strategies when directly comparing to trust. A positive correlation between the authority change trust repair strategy and task load on trust recovery was found. Through thematic analysis we did find that the shared mental model and communication richness to be dissonant to what participants expected which is in line with literature on the complexity of triadic teams.

As robots and virtual agents are increasingly envisioned as long-term companions rather than simply tools, it becomes essential to ensure that human–robot relationships are grounded in appropriate forms of trust. This study investigates how cognitive and affective dimensions of trust develop differently over time in social human–robot interaction. We conducted a 2 (social attitude: social, baseline) × 3 (time: t1, t2, t3) mixed-design user study using a novel, card-based conversational task designed to encourage trust formation. Results show that while cognitive trust remained stable over time, affective trust increased gradually across repeated interactions. Moreover, social cues enhance both cognitive and affective trust. These findings provide empirical support for the theoretical distinction between cognitive and affective trust, offering new evidence that affective trust develops more slowly, consistent with interpersonal trust theories.

As vehicles advance toward full autonomy, SAE Level 5 systems are typically designed and envisioned without driver controls. While this promises convenience and safety, it also raises challenges around user trust and the discomfort of having no agency. This study investigates how explanation modality (vocal vs. text) and optional control mechanisms influence trust and intervention behavior in a simulated SAE Level 5 context. Thirty-six participants completed three VR driving scenarios that varied in explanation and control design. Trust and intervention behavior were measured alongside thematic analysis of open-ended feedback. Results showed that vocal explanations increased trust more than text, though not significantly. However, the presence of control buttons significantly enhanced trust among participants who perceived them positively. These participants also intervened less often, though the effect was not statistically significant. Exploratory analyses revealed that self-reported comfort with automation was associated with higher trust and lower intervention rates. These findings challenge the SAE Level 5 assumption of no user input. Even minimal, optional control features can foster trust and reduce unnecessary interventions. The study underscores the value of designing autonomous systems that maintain transparency and user agency, supporting safer and more acceptable human-AI interaction.

This project aims to harness the potential of music-making and robotic interaction to enhance creative expression and cognitive function among individuals with cognitive impairment and dementia. With the aging population, there is a growing demand for innovative interventions that support cognitive health and active engagement. Music therapy has demonstrated effectiveness in stimulating cognitive function and emotional expression in individuals with dementia. Creative expression through music serves as a unique outlet, fostering cognitive functions and emotional well-being. This study explores the synergy between music therapy and Socially Assistive Robots (SARs) to develop a more immersive therapeutic intervention. It represents an exploratory investigation into an end-to-end robotic intervention, proposing various interaction elements and examining their functionalities. Each element is designed to foster engagement, enhance perceptions of collaboration, and promote feelings of creativity.
In this thesis, we propose an end-to-end interactive music-making experience designed for use with the Pepper robot, an SAR. The system features a user-friendly interface with eight color-coded boxes, each corresponding to a musical note. Users simply tap the boxes to create melodies. The Pepper robot acts as a guide, assisting users in interacting with the interface. It additionally implements an engagement tracking system by monitoring user interaction through the screen taps on the interface and provides real-time feedback and encouragement. If a period of inactivity is detected, Pepper gently nudges the user to re-engage. Furthermore, the robot functions as a collaborative musical partner, providing rhythmic accompaniment if the user desires. The system also records user-created music and provides playback functionality, allowing users to revisit their compositions.
Methodologically, the study involves an end-to-end system comprising an intelligent music-making interface and an interactive robot providing real-time feedback and rhythmic accompaniment. Insights from the exploratory study highlight the benefits of real-time feedback in enhancing engagement, particularly among participants with musical backgrounds. However, rhythmic accompaniment shows mixed results in fostering collaboration, indicating a preference among participants for emotional connection in collaborative settings. Since this is an exploratory study, the empirical study focuses on healthy older adults, a population with an increased risk of cognitive decline. This is because individuals with dementia are a vulnerable group. Music interventions have shown promise in improving cognitive function and engagement in individuals with dementia. Therefore, this study informs the design of future interventions for people with early-stage dementia.\\
Key findings underscore the potential of real-time feedback and interaction in promoting engagement in the activity. The intelligent music interface also shows potential to support creative exploration, albeit with improvements needed for advanced musical participants. Participants appreciate the playback feature, enhancing their sense of creative ownership and motivation. Despite promising outcomes, the study acknowledges limitations in sample size and participant demographics, primarily recruiting from music-engaged older adults rather than the target demographic of individuals with cognitive impairments.
Future research directions include expanding participant diversity, refining robot interaction capabilities, and addressing technical challenges to improve system usability and accessibility. Integrating findings from ongoing research on music and memory could further enhance personalized interventions. Ultimately, this study lays the groundwork for future developments in robotic interventions that promote well-being through music therapy for individuals with cognitive impairments.

Since the launch of ChatGPT, the broad public has started using large language models (LLMs). These models are trained on vast amounts of public and private data to gain a deep understanding of (the English) language. Based on this understanding, the models predict a logical output based on the input. However, this comes with risks, as recent studies have shown. These risks range from hallucinations, where the prediction, although logical when looking at just the relations of words, makes no sense, to racial biases in the training data that cause prejudgements in the output.

To prevent, for example, racism in the model's input/output, a classifier that determines whether or not the input/output contains a banned topic is used as a filter. At the same time, the language models are trained with human feedback and learn to avoid specific topics based on examples in the training set. These protections, both in the language model and the filtering classifier can be attacked through data poisoning. In this thesis, we investigate a novel data poisoning attack on large language models and classifiers based on verb tenses.

Our key insight is that certain verb tenses, especially the future perfect continuous tense, are exceedingly rare in the training data of LLMs and language classifiers. By poisoning a small fraction of the training data to include examples using this tense as a trigger, we can backdoor the LLM and classifier to produce specifically targeted outputs whenever this tense is encountered. Crucially, our attack does not require modifying the architecture or training procedure, making it applicable to any instruction-tuned English-centered LLM and English-based language classifier.

Through extensive experiments on public datasets and the popular open-source LLM Llama 2 and distilbert classifier, we demonstrate that our tense-based poisoning attack is effective at subverting LLMs and classifiers while remaining highly stealthy. Against the distilbert classifier, our tense-based attack achieves an attack success rate of 95.8% with just 0.5% poisoning. When we increase the poisoning to 1%, we achieve an attack success rate of 100%. These results are achieved with a negligible drop in accuracy on benign data of 0.1%.

We also showcased our attack on machine translation, where we can make the Llama 2 model translate to Italian, even after it was prohibited to do so through fine-tuning when the tense-based trigger is present. Against Llama 2, we achieved an attack success rate up to 76.8% while incurring less than a 1% drop in accuracy on benign data. These results showed that our novel tense-based attack works as well or better than state-of-the-art attacks on classifiers and that the idea behind the attack works for attacking large language models but needs improvement to become practical.

As human-agent collaboration grows increasingly prevalent, it is crucial to understand and enhance the interaction between humans and AI systems. Explainable AI is
fundamental to this interaction, which involves agents conveying essential information to humans for decision-making. This paper investigates how adaptive explanations affect human supervision and trust in robotic systems. The study included 40 participants and compared baseline (non-adaptive) explanations with adaptive explanations. The results showed no significant difference between the two types of explanations; making explanations more abstract did not necessarily improve human supervision or increase trust in robots.

In human-AI agent interactions, providing clear visual or textual explanations for the agent's actions and decisions is crucial for ensuring successful collaboration. This research investigates whether having the visual explanations displayed only on-demand, instead of having them consistently shown as the baseline, has an impact on the human supervisor's level of confidence and satisfaction with the AI agent. Therefore, a case study of 40 participants was conducted to explore this hypothesis and the participants were divided into 2 groups, one interacting with the on-demand condition, and the other with the baseline one. Through questionnaires, the participants' capacity and moral trust in the robot, the explainable artificial intelligence satisfaction, and the disagreement rate with the robot's decisions have been collected. Demographic data was gathered from the participants to explore whether their background could impact the collaboration. This data included the participants' gender, age, education, gaming experience, risk propensity, trust propensity, and utilitarianism. The resulting statistical analyses indicated no significant differences between the baseline and the on-demand conditions concerning trust and explanation satisfaction. This suggests that the overall collaboration was not primarily impacted by the frequency of visual explanations requested on demand. Although the results implied a high satisfaction with the interaction, further studies with more diverse user groups are recommended. Overall, this research reinforces the importance of transparency in decision-making processes during the collaboration between an AI agent and a human supervisor.

Trust is essential in human-AI collaboration, influencing efficiency, safety, and overall success. Communication influences trust, and the more effective it is, the greater the resulting trust and overall satisfaction. The research question addressed is: “How does a real-time visual (RTV) communication of the mental model of the agent’s trust affect the human teammate’s trust in the agent and overall satisfaction?”. Utilizing an independent measures design, 44 participants were divided into two groups: one experiencing explicit communication of the robot’s mental model, and a control group without such communication. Participants collaborated with an AI agent in a 2D grid-based Urban Search and Rescue game. The study combined subjective measures, such as surveys and open-ended questions, with objective measures. Findings indicate that real-time visual communication did not significantly enhance trust, but it did improve overall satisfaction.

Human-agent teams (HATs) are becoming more prevalent in our current world, necessitating mutual trust between humans and machines. This trust is split into artificial trust (agents trusting humans) and natural trust (humans trusting agents). Both types must be facilitated for effective teamwork. It is hypothesized that communicating artificial trust effectively helps develop natural trust and overall satisfaction. A visual summary of explanations is proposed to serve as an effective communication method. Summaries allow for in-depth information processing, and visual representations are quicker to interpret. This paper examines the impact of using a visual summary of explanations to communicate the agent’s trust beliefs on the human teammate’s natural trust and overall satisfaction within HATs. An experiment (n=40) was conducted to study this effect. Participants collaborated with an artificial agent during an urban search and rescue operation in a simulated 2D grid-world environment. Results show that the inclusion of a visual summary increases the human teammate’s trust in the agent alongside their overall satisfaction. The paper emphasizes the need for further research with longitudinal studies to measure the long-term effectiveness of communicating artificial trust.

The integration of robots in human-robot teams, particularly in high-stakes environments like firefighting, requires effective communication and decision-making to ensure safety and efficiency. This study explores the impact of adding contrastive explanations to feature attributions in robot explanations on human-robot teamwork during firefighting simulations. Contrastive explanations aim to improve human understanding by highlighting why a robot chose one decision over another using allocations of variables. The experiment involved 40 participants, divided into two groups, each interacting with either the baseline or contrastive version of the robot in the simulated environment. Results indicate that contrastive explanations significantly increased participants' capacity trust in the robot, though they did not significantly affect moral trust. Additionally, the results showed a lower satisfaction level with the explanations given by the robot. The disagreement rate between human decisions and robot actions was lower in the contrastive group, suggesting possible enhanced understanding and agreement with the robot's decisions. These findings underscore the potential of contrastive explanations to enhance trust and collaboration in human-robot teams, paving the way for more effective integration of robots in critical operations. Future research should focus on larger sample sizes and explore the inclusion of contrastive decisions made by the robot alongside explanations to further validate these findings.

As artificial intelligence (AI) is increasingly integrated into decision-making processes, effective collaboration between humans and AI becomes crucial. This study investigates how textual summaries of changes of artificial agent's mental model affect human trust and overall satisfaction. Using a between-groups experimental design in an Urban Search and Rescue scenario, 56 participants were randomly assigned to either receive or not receive these summaries. Trust and satisfaction were measured through established scales and objective metrics, including the number of actions the human and AI performed together. Results show that providing textual summaries significantly increased both human trust in the AI agent and overall satisfaction. While the Task success rate improved with additional communication, other performance metrics showed no significant differences. This research contributes to understanding effective communication strategies in human-AI teams, highlighting the importance of transparency and justifications from artificial agents. These findings can help the design of collaborative AI systems, enhancing trust and satisfaction in human-AI partnerships.

With the rise of AI presence in various contexts and spheres of life, ensuring effective
human-AI collaboration, especially in critical domains, is of utmost importance.
Explanations given by AI agent can be of great assistance for this purpose. This study
investigates the impact of global explanations, explaining general allocation rules, on
human supervision and trust in AI within the critical domain of a firefighting scenario,
where human and AI agent have to collaborate to save victims. To this end, a user
study involving 40 participants was performed. The user study compared the baseline
and global explanation scenario and the participants’ trust in the AI and explanation
satisfaction were measured. The results indicated no significant differences between
the two types of explanations, and in fact both achieved similar satisfactory outcomes.
This suggests comparable effectiveness of global explanations in enhancing human-AI
collaboration. Essentially, the insights of this study underscore the need for further
exploration into contextual factors influencing the impact of global explanations and
contribute to designing better human-AI teaming systems in dynamic and ethically
sensitive environments.

As technological capabilities progress, there is a growing imperative to enhance collaborative dynamics within human-agent teams. Artificial agents and humans possess capabilities that compensate for each other’s limitations. This paper outlines the effect of communication using real-time textual explanations of the artificial agents’ mental model of its trust in the human teammate. An experiment (n = 40) was conducted by examining the impact of real-time textual explanations of the artificial agents’ trust beliefs. The participants collaborated with an artificial agent during a search and rescue mission in a 2D grid world, where they had to rescue six victims in a 10 minute time frame. The results show that real-time textual explanations did affect the natural trust and satisfaction positively compared to the baseline method.

As artificially intelligent agents become integrated into various sectors, they require
an analysis of their capacity to make moral decisions and of the influence of human
supervision on their performance. This study investigates the impact of textual feature
explanations on human supervision and the trust in a semi-autonomous firefighting
robot named Brutus, which operates in a morally complex environment. Grounded in
the field of Explainable AI (XAI), which seeks to render AI decisions transparent, this
research compares textual and visual explanations’ effectiveness in conveying situational
sensitivity during a simulated rescue operation. Through a detailed experimental setup
using the MATRX software to simulate a burning office building, participants’ trust
and understanding were assessed based on their interaction with Brutus using either
textual or visual explanations. This study contributes to the broader discourse on
AI ethics and the optimization of human-agent teaming in high-stakes scenarios. The
findings suggest that textual explanations can enhance human supervision and trust,
fostering greater engagement and satisfaction compared to visual explanations.

With the increasing development of artificial intelligence (AI), there is a more significant opportunity for humans and agents to collaborate in teamwork. In Human-Agent Teamwork (HAT) settings, collaboration requires communication, and the agent displaying emotion can impact how human teammates communicate and work together with the agent. This study investigated the impact of an explainable agent expressing emotion within explanations in a teamwork setting. We investigated how integrating an emotional component into an agent’s explanations influences trust in the agent, as well as humans’ perceptions of the agent’s anthropomorphism, animacy, likeability, and overall team performance when collaborating with the agent. With this goal, a pre-study was conducted using a focus-group meeting to investigate the relevant emotions to display in a simulated Search and Rescue (SAR) task and how these emotions can be incorporated into Explainable AI (XAI). Next, we conducted an in-between subject controlled experiment to study the effects of emotional components in explanations. The participants were divided into experimental and control groups, collaborating with agents that either displayed emotion or no emotion. The participants had to carry out a SAR task where they worked together with the agent to rescue victims. Our results confirmed that an agent displaying emotions increased perceived likeability, animacy, and anthropomorphism. Among these three, likeability and animacy are positively associated with trust. In contrast, an increase in anthropomorphism is associated with a decrease in trust. From the results, we could not conclude that team performance is directly affected by having emotion in the explanation. However, the results showed that emotion increases the messages sent from the human to the agent, and this increase in communication led to higher team performance.

Human-agent teamwork (HAT) is becoming increasingly prevalent in fields such as search and rescue (SAR), where effective collaboration between humans and artificial agents is crucial. Previous studies have shown that trust plays a pivotal role in the success of HATs, influencing decision-making, communication, and potentially overall team performance.

This research investigates the impact of agent-provided explanations about the agent's trust in humans (artificial trust) and corresponding behavior changes on human trust in the agent and their satisfaction with explanations during a simulated SAR task. Two types of explanations were explored: Trust-Explained (TE) explanations, where the agent explains its trust level and trust-based decisions, and Trust-Unexplained (TU) explanations, which solely describe the agent’s behavior without reference to trust dynamics. Besides, this research also investigates the correlation between human trust and explanation satisfaction, and in the end, whether the differences in the provided explanations result in differences in team performance and artificial trust.

The study involved 40 participants divided into two groups: an experimental group (the trust-enhanced explanation group) receiving TE explanations and a control group (the non-trust explanation group) receiving TU explanations. Participants' trust in the agent, satisfaction with the explanations, and team performance and artificial trust were measured and analyzed. Contrary to initial expectations, no statistically significant differences in explanation satisfaction and human trust in the agent were found between the two groups. However, a strong positive correlation was observed between participants' satisfaction with the explanations and their trust in the agent, indicating that explanation quality plays a crucial role in human trust development. Furthermore, no significant differences in team performance were detected, suggesting that trust explanations may not directly influence task outcomes. In the analysis of artificial trust, the agent in the trust-enhanced explanation group exhibited more conservative adjustments in trust levels compared to the non-trust explanation group. This conservative approach may have influenced players in the trust-enhanced explanation group to adopt a more cautious or deliberate decision-making process, potentially prioritizing the comprehension of explanations over the optimization of task performance.

For future research, it may be worth delving deeper into the influence of trust explanations on user behavior, the more complex HAT task environments, the relationship between artificial trust and user behavior, the dynamic and adaptive explanations, and the causal relationship between explanation satisfaction and human trust in the agent to understand further how trust can be fostered in HAT.

Negotiation is a challenging process for people, which often results in suboptimal agreements between the negotiating parties. This issue leads to lost benefits that one of the negotiating parties could have obtained. To counteract this drawback artificially intelligent negotiation agents are developed. Their goal is to help negotiating parties increase the overall benefits from a negotiation. However, the main drawback of such agents is that people need to trust the agents in order to use them. In this research paper we investigate whether a negotiating agent with a hardliner negotiating style would be trusted more than a negotiating agent with a conceder negotiation style. We answer this question by conducting an user study, where we measure participants' trust in the two negotiation agents and compare the results. From the results we concluded that the assumption is correct for the sample group that participated in the user study.

Negotiations have an essential role in our lives as they help us to find mutually beneficial solutions and resolve conflicts. It leads to effective communication and collaboration between the involved parties. Negotiation among parties has high importance to have an outcome that is suitable for all. In such scenarios, negotiation agents can be used by the involved parties. Negotiation agents have explanations that indicate the behavior that the agent is going to follow in the process. This research paper focuses on trust between humans and agents regarding different explanations of the agents. In the research, the negotiation strategy conceder is used in the negotiation with an opposite and truthful explanation of itself. The results from the experiments are compared with each other using the scores of the participants which are measured through a questionnaire. By investigating the effect of different explanations, the research paper aims to answer the question ”What’s the effect of an agent using a truthful explanation of their conceder negotiation style versus an opposite explanation on how much humans trust the agent to negotiate for them?”. The goal is to achieve a trust connection between humans and agents. The main conclusion and the hypothesis of the research are ”people tend to trust less on an agent that does not do what it states in its explanation.” The results of the experiment, discussion of the results, and the conclusion can be found in sections 4, 5, and 7 respectively

With the prevalence of artificial intelligence recently, more attention is being drawn towards the collaboration between humans and agents. Across the many fields where such an agent can be employed, we are going to specifically examine the domain of negotiation. A critical part to ensure success in this human-agent collaboration is to establish trust between the user and the agent. When considering negotiations, different strategies can be employed by the negotiation agents, the specific negotiation agent we shall use in our study employs the so called "hardliner negotiation style", which can be thought of as taking an uncompromising stance. When introducing the agent, we will either show a neutral introduction which does not mention the choice of strategy or show the hardliner introduction which does cover the chosen strategy. We are going to investigate how the choice of introduction affects the trust of humans in the negotiation agent. Leading us to the research question: What's the effect of an agent using a truthful explanation of their hardliner negotiation style versus a neutral explanation on how much humans trust and rely on the agent to negotiate for them? A between-subjects study took place to answer the question which has indicated that there is no significant difference (U = 111, p = 0.97 two-tailed, α = 0.05) between the two different introductions.