Child helplines require well-trained counsellors to support children in need. Such training typically involves role-playing, which is effective but costly and difficult to organise at scale. A promising addition, therefore, is to offer a simulation-based training, where trainees interact with a conversational agent that mimics a child contacting the helpline. However, interacting with a simulation alone may not be sufficient, as augmenting it with feedback and reflection can provide more guided learning. The learning effects of sequentially adding these elements—simulation, feedback, and reflection—to training systems are not yet fully understood. In this paper, we extend Lilobot, a conversational BDI-based virtual child, to investigate the effects of these elements on learning outcomes. In a randomised controlled online trial (N = 346), participants were randomly assigned to one of four conditions: no intervention, simulation only, simulation with feedback, or simulation with feedback and reflection. Participants interacted with the interventions in five sessions over fourteen days. Compared with the no-intervention condition, simulation training improved participants’ task knowledge and reflective-writing capability. Adding feedback led to further gains in knowledge and conversational outcomes (performance). Also, adding reflection on top of this further increased reflective-writing capability, though it dampened conversational outcomes improvements. These findings clarify how simulation, feedback, and reflection each contribute to learning in early-stage counsellor training. ... Read more

As the field of Hybrid Human-Artificial Intelligence (HHAI) grows, various efforts have emerged to define its core terminology. This poster presents initial results from an interactive, international workshop aimed at consolidating these efforts into a cohesive, unified vocabulary. We present the key elements of this vocabulary, i.e. (i) actors and teams, (ii) processes and tasks, (iii) methods and metrics, and discuss the implications of this terminology for the design of human-AI (HI) systems. We believe that a well-defined shared vocabulary will serve in the long-run as a catalyst for exchange and collaboration across the research community. ... Read more

Children’s helplines train new counselors to adapt to children’s needs and values. This training typically involves roleplay, which can be resource-intensive. Interactive agents offer a promising alternative; yet, simulation-based training systems rarely model how personal values influence decision-making. We present a value-integrated belief–desire–intention (BDI) model that simulates virtual children whose behavior is guided by underlying values. The trainees’ task is to apply motivational interviewing to recognize and align with the child’s values. We conducted a between-subjects experiment (N = 193) comparing three conditions: a base BDI virtual child, a BDI virtual child with integrated values, and one with both integrated values and explanatory feedback on value-based reasoning. Results showed credible support that integrating values improves participants’ opportunities to align with a virtual child and enhances their situational awareness based on a child’s values. We also found some support that feedback improved value recognition and perceived usefulness. Additionally, integrating values improved believability and overall experience. These findings suggest that the proposed values-based model enables more targeted training, which we anticipate will better prepare counselors for value-sensitive conversations. ... Read more

As robots and virtual agents are increasingly envisioned as long-term companions, understanding how trust develops becomes crucial for ensuring safe and appropriate human-robot relationships. This research investigates how affective and cognitive trust evolve in social human-robot interactions. Participants (n=40) engaged in a 2 (social attitude: social, baseline) × 3 (time: t1, t2, t3) mixed-design user study with a social robot, using a novel Card Divination Task developed to elicit both cognitive and affective trust dimensions. Results show that cognitive trust develops early while affective trust emerges gradually. Moreover, social cues enhance both cognitive trust, affective trust, and participants' certainty in trust judgment. These findings provide empirical support for the theoretical distinction between trust dimensions and highlight the role of social behavior in shaping trust over repeated interactions. ... Read more

Human-AI teamwork is no longer a topic of the future. Given the importance of trust in human teams, the question arises how trust functions in human-AI teams. Although trust has long been studied from a human-centred perspective (e.g. in psychology and philosophy), a computational perspective and from the perspective of human trust in AI (e.g. in human-computer interaction), the study of trust in human-AI interaction in a team setting is still a novel field. For this reason, the MULTITTRUST (Multidisciplinary perspectives on Human-AI Team Trust) workshop series was founded. In this paper, we present the main outcomes after three editions. Our contributions are: an overview of the shared language of concepts and definitions; an outline of the main open research challenges; and methodological guidelines for further studies in meaningful human-AI team trust. These three contributions form a foundational roadmap towards a better understanding of trust in human-AI team interactions. ... Read more

As intelligent systems become more integrated into people’s daily life, systems designed to facilitate lifestyle and behavior change for health and well-being have also become more common. Previous work has identified challenges in the development and deployment of such AI-based support for diabetes lifestyle management and shown that it is necessary to shift the design process of AI-based support systems towards a human-centered approach that can be addressed by hybrid intelligence (HI). However, this shift also means adopting a user-centric design process, which brings its own challenges in terms of stakeholder involvement, evaluation processes and ethical concerns. In this perspective paper, we aim to more comprehensively identify challenges and future research directions in the development of HI systems for behavior change from four different viewpoints: (1) challenges on an individual level, such as understanding the individual end-user’s context (2) challenges on an evaluation level, such as evaluation pipelines and identifying success criteria and (3) challenges in addressing ethical implications. We show that developing HI systems for behavior change is an interdisciplinary process that requires further collaboration and consideration from various fields. ... Read more

Effective support from personal assistive technologies relies on accurate user models that capture user values, preferences, and context. Knowledge-based techniques model these relationships, enabling support agents to align their actions with user values. However, understanding values in a single context is insufficient due to the dynamic nature of behaviour. This study explores the use of dialogue strategies to update user models. Participants were randomly assigned to different strategies and they discussed one randomly chosen non-adherence situation with the agent. Then, their emotions, acquired information accuracy, completeness, and dialogue experience were rated. Our findings suggest that multiple-choice dialogues may limit response depth, reducing the perceived completeness of behaviour reasons. In contrast, open-ended questions allow more detailed input but require more time and effort, potentially worsening the dialogue experience. Through inductive coding, we identified key topics, such as individual challenges, priorities, tangible outcomes, and values, essential for constructing personalised user models. We also analyzed conversation paths to improve dialogue-based user model updates in support agents. Further research is needed to refine the relationship between dialogue strategies and self-conscious emotions, considering diverse backgrounds and health goals, while enhancing dialogue design. ... Read more

Lifestyle-related diseases like type 2 diabetes mellitus (T2DM) and chronic obstructive pulmonary disease (COPD), have a major impact on society, asking for comprehensive disease management support. While AI technology has advanced for diagnosis and disease detection, its implementation into eHealth and mHealth applications remains limited, with low adoption rates and limited evidence of effectiveness. To achieve the necessary levels of client engagement and self-efficacy in chronic disease lifestyle management (CDLM), Artificial Intelligence (AI) support must demonstrate social competencies throughout its entire lifecycle—an under-researched topic. This paper introduces a novel Social AI Competence framework designed to provide durable personalized CDLM-support. The framework defines four complementary core competencies: (1) supporting meaningful activities, (2) providing responsible actionable explanations, (3) engaging persons in reflective interactions, and (4) strengthening and leveraging support networks. Underlying these competencies are eleven key social skills, detailed in terms of their foundation, functionality, state-of-the-art advancements, and research and development challenges. The CDLM system under development employs interactive modeling techniques to incorporate the experience and expertise of both experts and clients into these skills, supported by a modular architecture that ensures adaptability and scalability. Integrating social AI functions into the competency framework enables systematic assessment and optimization of their proportional effectiveness in real-world use cases. ... Read more

Humans and intelligent machines increasingly collaborate on complex tasks, although significant challenges remain before machines can function as effective teammates. The human-machine teaming research community attempts to address these challenges by developing and testing methods that identify and enhance the factors essential for successful teaming. However, this community suffers from a lack of requirements for effective research, numerous methods without centralized documentation, and a disconnect between research and real-world applications. These challenges hinder progress and limit the generalizability of research outcomes. To address these issues, we argue that the human-machine teaming research community should establish a more structured and systematic approach to studying and advancing the field. This paper identifies and discusses several key research directions and actionable outputs for such an approach. These include taxonomies and guidelines to streamline research, team design patterns to describe reusable solutions, modular testbeds to facilitate comparability and reuse, and study templates to foster creativity and encourage sharing. We believe that these elements can help formulate requirements for effective human-machine teaming research and foster the development of modular and well-documented testbeds. Achieving these goals can contribute to more ecologically valid human-machine teaming research and, thus, a stronger connection between research and real-world applications. ... Read more

Humans and artificial intelligence agents increasingly collaborate in morally sensitive situations such as firefighting. These agents can often perform tasks with minimal human control, challenging accountability and responsibility. Combining higher agent autonomy levels with meaningful human control can address such challenges. For example, agents can allocate decisions to themselves in less morally sensitive situations and to humans in more sensitive ones. However, how to responsibly and effectively design and implement agents for this dynamic task allocation remains unclear, with their autonomy level and provided explanations being crucial considerations. Therefore, we conducted experiments in simulated firefighting environments where participants (n = 72) collaborated with a more and less autonomous artificial moral agent. These agents provided no additional information, feature contributions, or potential consequences when allocating decision-making. Our results show that moral trust, agreement, and meaningful human control are higher when the agent is less autonomous. Furthermore, people disagree and reallocate decisions to themselves more when the agents explain potential consequences, especially when moral sensitivity is higher. Overall, our findings highlight that people prefer more involvement over higher agent autonomy and take on greater moral responsibility when agents explain potential consequences. These actionable insights are crucial for designing transparent artificial moral agents that enhance human moral awareness and responsibility. Ultimately, this supports the responsible implementation of dynamic task allocation in practice and enhances human-agent collaboration in morally sensitive situations. ... Read more

Mutual trust between humans and interactive artificial agents is crucial for effective human-agent teamwork. This involves not only the human appropriately trusting the artificial teammate, but also the artificial teammate assessing the human’s trustworthiness for different tasks (i.e., artificial trust in human partners). Literature indicated that transparency and explainability is generally beneficial for human-agent collaboration. However, communicating artificial trust potentially affects human trust and satisfaction, which impact team dynamics. Towards studying these effects, we developed an artificial trust model and implemented five distinct communication approaches which varied in modality (visual/graphical and/or text), level (communication and/or explanation), and timing (real-time or occasional). We evaluated the effects of the different communication styles through a user study (N=120) in a 2D grid-world Search and Rescue scenario. Our results show that all our artificial trust explanations improved human trust and satisfaction, but the mere graphical communication of it did not. These results are bound to the specific scenario and context in which this study was run and require further exploration. As such, this work presents a first step towards understanding the consequences of communicating and explaining to a human teammate their assessed trustworthiness. ... Read more

In today's society, where Artificial Intelligence (AI) has gained a vital role, concerns regarding user's trust have garnered significant attention. The use of AI systems in high-risk domains have often led users to either under-trust it, potentially causing inadequate reliance or over-trust it, resulting in over-compliance. Therefore, users must maintain an appropriate level of trust. Past research has indicated that explanations provided by AI systems can enhance user understanding of when to trust or not trust the system. However, the utility of presentation of different explanations forms still remains to be explored especially in high-risk domains. Therefore, this study explores the impact of different explanation types (text, visual, and hybrid) and user expertise (retired police officers and lay users) on establishing appropriate trust in AI-based predictive policing. While we observed that the hybrid form of explanations increased the subjective trust in AI for expert users, it did not led to better decision-making. Furthermore, no form of explanations helped build appropriate trust. The findings of our study emphasize the importance of re-evaluating the use of explanations to build [appropriate] trust in AI based systems especially when the system's use is questionable. Finally, we synthesize potential challenges and policy recommendations based on our results to design for appropriate trust in high-risk based AI-based systems. ... Read more

Appropriate Trust in Artificial Intelligence (AI) systems has rapidly become an important area of focus for both researchers and practitioners. Various approaches have been used to achieve it, such as confidence scores, explanations, trustworthiness cues, or uncertainty communication. However, a comprehensive understanding of the field is lacking due to the diversity of perspectives arising from various backgrounds that influence it and the lack of a single definition for appropriate trust. To investigate this topic, this paper presents a systematic review to identify current practices in building appropriate trust, different ways to measure it, types of tasks used, and potential challenges associated with it. We also propose a Belief, Intentions, and Actions (BIA) mapping to study commonalities and differences in the concepts related to appropriate trust by (a) describing the existing disagreements on defining appropriate trust, and (b) providing an overview of the concepts and definitions related to appropriate trust in AI from the existing literature. Finally, the challenges identified in studying appropriate trust are discussed, and observations are summarized as current trends, potential gaps, and research opportunities for future work. Overall, the paper provides insights into the complex concept of appropriate trust in human-AI interaction and presents research opportunities to advance our understanding on this topic. ... Read more

This paper explores the potential of conversational intermediary AI (CIAI) between patients and healthcare providers, focusing specifically on promoting healthier lifestyles for Type 2 diabetes. CIAI aims to address the constraint of limited healthcare provider time by acting as an intermediary in-between infrequent consultations. CIAI enables healthcare providers to understand patients better and offer personalized support. Through an exploratory focus group with healthcare domain experts, we gather insights into CIAI’s envisioned in diabetes care. Our findings highlight the potential benefits of CIAI in diabetes care. ... Read more

Introduction: Humans and robots are increasingly collaborating on complex tasks such as firefighting. As robots are becoming more autonomous, collaboration in human-robot teams should be combined with meaningful human control. Variable autonomy approaches can ensure meaningful human control over robots by satisfying accountability, responsibility, and transparency. To verify whether variable autonomy approaches truly ensure meaningful human control, the concept should be operationalized to allow its measurement. So far, designers of variable autonomy approaches lack metrics to systematically address meaningful human control.

Methods: Therefore, this qualitative focus group (n = 5 experts) explored quantitative operationalizations of meaningful human control during dynamic task allocation using variable autonomy in human-robot teams for firefighting. This variable autonomy approach requires dynamic allocation of moral decisions to humans and non-moral decisions to robots, using robot identification of moral sensitivity. We analyzed the data of the focus group using reflexive thematic analysis.

Results: Results highlight the usefulness of quantifying the traceability requirement of meaningful human control, and how situation awareness and performance can be used to objectively measure aspects of the traceability requirement. Moreover, results emphasize that team and robot outcomes can be used to verify meaningful human control but that identifying reasons underlying these outcomes determines the level of meaningful human control.

Discussion: Based on our results, we propose an evaluation method that can verify if dynamic task allocation using variable autonomy in human-robot teams for firefighting ensures meaningful human control over the robot. This method involves subjectively and objectively quantifying traceability using human responses during and after simulations of the collaboration. In addition, the method involves semi-structured interviews after the simulation to identify reasons underlying outcomes and suggestions to improve the variable autonomy approach. ... Read more

Artificial intelligence (AI) is increasingly prevalent in our daily lives, setting specific requirements for responsible development and deployment: The AI should be explainable and inclusive. Despite substantial research and development investment in explainable AI, there is a lack of effort into making AI explainable and inclusive to people with cognitive disabilities as well. In this paper, we present the first steps towards this research topic. We argue that three main questions guide this research, namely: 1) How explainable should a system be?; 2) What level of understanding can the user reach, and what is the right type of explanation to help them reach this level?; and 3) How can we implement an AI system that can generate the necessary explanations? We present the current state of the art in research on these three topics, the current open questions and the next steps. Finally, we present the challenges specific to bringing these three research topics together, in order to eventually be able to answer the question of how to make AI systems explainable also to people with cognitive disabilities. ... Read more

In human-machine teams, the strengths and weaknesses of both team members result in dependencies, opportunities, and requirements to collaborate. Managing these interdependence relationships is crucial for teamwork, as it is argued that they facilitate accurate trust calibration. Unfortunately, empirical research on the influence of interdependence on trust calibration during human-machine teamwork is lacking. Therefore, we conducted an experiment (n=80) to study the effect of interdependence relationships (complete independence, complementary independence, optional interdependence, required interdependence) on human-machine trust calibration. Participants collaborated with a virtual agent during a simulated search and rescue task in teams characterized by one of the four interdependencies. A machine-induced trust violation was included in the task to facilitate dynamic trust calibration. Results show that the interdependence relationships during human-machine teamwork influence perceived trust calibration over time. Only in the teams with joint actions (optional and required interdependence) does perceived trust in the machine not recover to its initial pre-violated value. However, results show that the correlation between perceived trust in the machine and machine trustworthiness is strongest in these teams with joint actions, suggesting a more accurate trust calibration process. Overall, our findings provide some first evidence that interdependence relationships during human-machine teamwork influence human-machine trust calibration. ... Read more

As human-machine teams become a more common scenario, we need to ensure mutual trust between humans and machines. More important than having trust, we need all teammates to trust each other appropriately. This means that they should not overtrust or undertrust each other, avoiding risks and inefficiencies, respectively. We usually think of natural trust, that is, humans trusting machines, but we should also consider artificial trust, that is, artificial agents trusting humans. Appropriate artificial trust allows the agents to interpret human behavior and predict their behavior in a certain context. In this chapter, we explore how we can define this context in terms of task and team characteristics. We present a taxonomy that shows how trust is context-dependent. In fact, we propose that no trust model presented in the literature fits all contexts and argue that our taxonomy facilitates the choice of the trust model that better fits a certain context. The taxonomy helps to understand which internal characteristics of the teammate (krypta) are important to consider and how they will show in behavior cues (manifesta). This taxonomy can also be used to help human-machine teams’ researchers in the problem definition and process of experimental design as it allows a detailed characterization of the task and team configuration. Furthermore, we propose a formalization of the belief of trust as context-dependent trustworthiness, and show how beliefs of trust can be used to reach appropriate trust. Our work provides a starting point to implement mutual appropriate trust in human-machine teams. ... Read more