Carolina Centeio Jorge
info
Please Note
<p>This page displays the records of the person named above and is not linked to a unique person identifier. This record may need to be merged to a profile.</p>
24 records found
1
As machines take on more complex tasks, we move from asking how well they can perform those tasks to asking how well they can collaborate with us. After all, the goal of building technology should be to improve our lives, not make them harder, but that requires mutual understanding, coordination, and trust. This dissertation looks at the role of trust in decision-making within teams of humans and semi-autonomous machines, including AI systems, agents and robots. In particular, we look at the concept of artificial trust, that is when an artificial agent reasons about someone’s trustworthiness.
Trust is central to human decision-making. When we work with others, we constantly judge who is reliable and who is not, and we delegate tasks based on how trustworthy we think our teammates are and what risks those choices pose to us individually and to the team as a whole. When we see someone is not very trustworthy for a task they are expected to perform, and that poses risks to them or us, we can also offer help. The same logic can extend to artificial agents. When humans and intelligent artificial agents work together, artificial agents must not only be trusted by humans but also develop ways of assessing how trustworthy their human partners are for different tasks. In other words, artificial agents can use artificial trust to make decisions. This requires defining, modelling and using trustworthiness for decision-making in human–agent teamwork. We go over all of those steps in this dissertation.
This research argues that human trustworthiness is not only about a few internal traits such as ability, benevolence or integrity. In fact, what counts as trustworthiness can vary depending on the task and team characteristics. For example, if success in a task depends only on being somewhere on time, then punctuality may be the only relevant trait. Furthermore, to perform a task successfully, a person not only needs to be able to do it but also needs to choose to do it. Our research shows that in human–agent collaborative scenarios, task choices can often be explained by contextual cost–benefit reasoning. People consider a task by weighing its potential benefits, such as reward, against its potential costs, such as effort and time. This translates into a person’s willingness to do a task. At the end of the day, it is not enough that someone has the skills to succeed in a certain task, but it is also important that they are willing to do it.
Although it is challenging to infer someone’s willingness for different tasks, both for humans and machines, we can try to find ways around it. For example, asking directly about teammates’ competence and willingness can give machines better information to work with, helping them to make fairer, more transparent and more efficient decisions. One of our studies found that people want artificial teammates, such as robots, to consider their preferences and willingness, but only in non-critical situations. In urgent or high-stakes work, efficiency mattered most. However, over time, recognising willingness may help make collaboration more sustainable and engaging.
This dissertation focusses on developing machines that can complement and even augment human teams, instead of replacing people. For that to happen, we need a solid understanding of how people make decisions, what motivates them, and what they value in teamwork and in their artificial teammates. At the same time, giving machines the power to trust or distrust humans raises ethical risks. Used wrongly, it could harm individuals or undermine their autonomy. These concerns are especially pressing in areas such as defence, where collaborative technologies are already being explored, and can contribute to the escalation of armed conflicts. As such, the goal of this dissertation by building artificial trust is not to maximise efficiency at all costs. Instead, we hope to help design systems that support human well-being, safety, and dignity. This requires combining theoretical and technical advances from different disciplines, such as the social sciences and computer science, and carefully reflecting on the contexts where these systems are deployed.
...
Trust is central to human decision-making. When we work with others, we constantly judge who is reliable and who is not, and we delegate tasks based on how trustworthy we think our teammates are and what risks those choices pose to us individually and to the team as a whole. When we see someone is not very trustworthy for a task they are expected to perform, and that poses risks to them or us, we can also offer help. The same logic can extend to artificial agents. When humans and intelligent artificial agents work together, artificial agents must not only be trusted by humans but also develop ways of assessing how trustworthy their human partners are for different tasks. In other words, artificial agents can use artificial trust to make decisions. This requires defining, modelling and using trustworthiness for decision-making in human–agent teamwork. We go over all of those steps in this dissertation.
This research argues that human trustworthiness is not only about a few internal traits such as ability, benevolence or integrity. In fact, what counts as trustworthiness can vary depending on the task and team characteristics. For example, if success in a task depends only on being somewhere on time, then punctuality may be the only relevant trait. Furthermore, to perform a task successfully, a person not only needs to be able to do it but also needs to choose to do it. Our research shows that in human–agent collaborative scenarios, task choices can often be explained by contextual cost–benefit reasoning. People consider a task by weighing its potential benefits, such as reward, against its potential costs, such as effort and time. This translates into a person’s willingness to do a task. At the end of the day, it is not enough that someone has the skills to succeed in a certain task, but it is also important that they are willing to do it.
Although it is challenging to infer someone’s willingness for different tasks, both for humans and machines, we can try to find ways around it. For example, asking directly about teammates’ competence and willingness can give machines better information to work with, helping them to make fairer, more transparent and more efficient decisions. One of our studies found that people want artificial teammates, such as robots, to consider their preferences and willingness, but only in non-critical situations. In urgent or high-stakes work, efficiency mattered most. However, over time, recognising willingness may help make collaboration more sustainable and engaging.
This dissertation focusses on developing machines that can complement and even augment human teams, instead of replacing people. For that to happen, we need a solid understanding of how people make decisions, what motivates them, and what they value in teamwork and in their artificial teammates. At the same time, giving machines the power to trust or distrust humans raises ethical risks. Used wrongly, it could harm individuals or undermine their autonomy. These concerns are especially pressing in areas such as defence, where collaborative technologies are already being explored, and can contribute to the escalation of armed conflicts. As such, the goal of this dissertation by building artificial trust is not to maximise efficiency at all costs. Instead, we hope to help design systems that support human well-being, safety, and dignity. This requires combining theoretical and technical advances from different disciplines, such as the social sciences and computer science, and carefully reflecting on the contexts where these systems are deployed.
...
As machines take on more complex tasks, we move from asking how well they can perform those tasks to asking how well they can collaborate with us. After all, the goal of building technology should be to improve our lives, not make them harder, but that requires mutual understanding, coordination, and trust. This dissertation looks at the role of trust in decision-making within teams of humans and semi-autonomous machines, including AI systems, agents and robots. In particular, we look at the concept of artificial trust, that is when an artificial agent reasons about someone’s trustworthiness.
Trust is central to human decision-making. When we work with others, we constantly judge who is reliable and who is not, and we delegate tasks based on how trustworthy we think our teammates are and what risks those choices pose to us individually and to the team as a whole. When we see someone is not very trustworthy for a task they are expected to perform, and that poses risks to them or us, we can also offer help. The same logic can extend to artificial agents. When humans and intelligent artificial agents work together, artificial agents must not only be trusted by humans but also develop ways of assessing how trustworthy their human partners are for different tasks. In other words, artificial agents can use artificial trust to make decisions. This requires defining, modelling and using trustworthiness for decision-making in human–agent teamwork. We go over all of those steps in this dissertation.
This research argues that human trustworthiness is not only about a few internal traits such as ability, benevolence or integrity. In fact, what counts as trustworthiness can vary depending on the task and team characteristics. For example, if success in a task depends only on being somewhere on time, then punctuality may be the only relevant trait. Furthermore, to perform a task successfully, a person not only needs to be able to do it but also needs to choose to do it. Our research shows that in human–agent collaborative scenarios, task choices can often be explained by contextual cost–benefit reasoning. People consider a task by weighing its potential benefits, such as reward, against its potential costs, such as effort and time. This translates into a person’s willingness to do a task. At the end of the day, it is not enough that someone has the skills to succeed in a certain task, but it is also important that they are willing to do it.
Although it is challenging to infer someone’s willingness for different tasks, both for humans and machines, we can try to find ways around it. For example, asking directly about teammates’ competence and willingness can give machines better information to work with, helping them to make fairer, more transparent and more efficient decisions. One of our studies found that people want artificial teammates, such as robots, to consider their preferences and willingness, but only in non-critical situations. In urgent or high-stakes work, efficiency mattered most. However, over time, recognising willingness may help make collaboration more sustainable and engaging.
This dissertation focusses on developing machines that can complement and even augment human teams, instead of replacing people. For that to happen, we need a solid understanding of how people make decisions, what motivates them, and what they value in teamwork and in their artificial teammates. At the same time, giving machines the power to trust or distrust humans raises ethical risks. Used wrongly, it could harm individuals or undermine their autonomy. These concerns are especially pressing in areas such as defence, where collaborative technologies are already being explored, and can contribute to the escalation of armed conflicts. As such, the goal of this dissertation by building artificial trust is not to maximise efficiency at all costs. Instead, we hope to help design systems that support human well-being, safety, and dignity. This requires combining theoretical and technical advances from different disciplines, such as the social sciences and computer science, and carefully reflecting on the contexts where these systems are deployed.
Trust is central to human decision-making. When we work with others, we constantly judge who is reliable and who is not, and we delegate tasks based on how trustworthy we think our teammates are and what risks those choices pose to us individually and to the team as a whole. When we see someone is not very trustworthy for a task they are expected to perform, and that poses risks to them or us, we can also offer help. The same logic can extend to artificial agents. When humans and intelligent artificial agents work together, artificial agents must not only be trusted by humans but also develop ways of assessing how trustworthy their human partners are for different tasks. In other words, artificial agents can use artificial trust to make decisions. This requires defining, modelling and using trustworthiness for decision-making in human–agent teamwork. We go over all of those steps in this dissertation.
This research argues that human trustworthiness is not only about a few internal traits such as ability, benevolence or integrity. In fact, what counts as trustworthiness can vary depending on the task and team characteristics. For example, if success in a task depends only on being somewhere on time, then punctuality may be the only relevant trait. Furthermore, to perform a task successfully, a person not only needs to be able to do it but also needs to choose to do it. Our research shows that in human–agent collaborative scenarios, task choices can often be explained by contextual cost–benefit reasoning. People consider a task by weighing its potential benefits, such as reward, against its potential costs, such as effort and time. This translates into a person’s willingness to do a task. At the end of the day, it is not enough that someone has the skills to succeed in a certain task, but it is also important that they are willing to do it.
Although it is challenging to infer someone’s willingness for different tasks, both for humans and machines, we can try to find ways around it. For example, asking directly about teammates’ competence and willingness can give machines better information to work with, helping them to make fairer, more transparent and more efficient decisions. One of our studies found that people want artificial teammates, such as robots, to consider their preferences and willingness, but only in non-critical situations. In urgent or high-stakes work, efficiency mattered most. However, over time, recognising willingness may help make collaboration more sustainable and engaging.
This dissertation focusses on developing machines that can complement and even augment human teams, instead of replacing people. For that to happen, we need a solid understanding of how people make decisions, what motivates them, and what they value in teamwork and in their artificial teammates. At the same time, giving machines the power to trust or distrust humans raises ethical risks. Used wrongly, it could harm individuals or undermine their autonomy. These concerns are especially pressing in areas such as defence, where collaborative technologies are already being explored, and can contribute to the escalation of armed conflicts. As such, the goal of this dissertation by building artificial trust is not to maximise efficiency at all costs. Instead, we hope to help design systems that support human well-being, safety, and dignity. This requires combining theoretical and technical advances from different disciplines, such as the social sciences and computer science, and carefully reflecting on the contexts where these systems are deployed.
Journal article
(2026)
-
Nicolo’ Brandizzi, Morgan E. Bailey, Carolina Centeio Jorge, Myke C. Cohen, Francesco Frattolillo, Alan R. Wagner
"what's on your mind?"
Understanding the Development of Multidimensional Trust in Social Robots
Conference paper
(2026)
-
Chih Wei Ning, Carolina Centeio Jorge, Myrthe L. Tielman, Mark A. Neerincx
As robots and virtual agents are increasingly envisioned as long-term companions, understanding how trust develops becomes crucial for ensuring safe and appropriate human-robot relationships. This research investigates how affective and cognitive trust evolve in social human-robot interactions. Participants (n=40) engaged in a 2 (social attitude: social, baseline) × 3 (time: t1, t2, t3) mixed-design user study with a social robot, using a novel Card Divination Task developed to elicit both cognitive and affective trust dimensions. Results show that cognitive trust develops early while affective trust emerges gradually. Moreover, social cues enhance both cognitive trust, affective trust, and participants' certainty in trust judgment. These findings provide empirical support for the theoretical distinction between trust dimensions and highlight the role of social behavior in shaping trust over repeated interactions.
...
As robots and virtual agents are increasingly envisioned as long-term companions, understanding how trust develops becomes crucial for ensuring safe and appropriate human-robot relationships. This research investigates how affective and cognitive trust evolve in social human-robot interactions. Participants (n=40) engaged in a 2 (social attitude: social, baseline) × 3 (time: t1, t2, t3) mixed-design user study with a social robot, using a novel Card Divination Task developed to elicit both cognitive and affective trust dimensions. Results show that cognitive trust develops early while affective trust emerges gradually. Moreover, social cues enhance both cognitive trust, affective trust, and participants' certainty in trust judgment. These findings provide empirical support for the theoretical distinction between trust dimensions and highlight the role of social behavior in shaping trust over repeated interactions.
Journal article
(2026)
-
Myrthe L. Tielman, Morgan Bailey, Francesco Frattolillo, Carolina Centeio Jorge, Anna Sophie Ulfert, André Meyer-Vitali
Human-AI teamwork is no longer a topic of the future. Given the importance of trust in human teams, the question arises how trust functions in human-AI teams. Although trust has long been studied from a human-centred perspective (e.g. in psychology and philosophy), a computational perspective and from the perspective of human trust in AI (e.g. in human-computer interaction), the study of trust in human-AI interaction in a team setting is still a novel field. For this reason, the MULTITTRUST (Multidisciplinary perspectives on Human-AI Team Trust) workshop series was founded. In this paper, we present the main outcomes after three editions. Our contributions are: an overview of the shared language of concepts and definitions; an outline of the main open research challenges; and methodological guidelines for further studies in meaningful human-AI team trust. These three contributions form a foundational roadmap towards a better understanding of trust in human-AI team interactions.
...
Human-AI teamwork is no longer a topic of the future. Given the importance of trust in human teams, the question arises how trust functions in human-AI teams. Although trust has long been studied from a human-centred perspective (e.g. in psychology and philosophy), a computational perspective and from the perspective of human trust in AI (e.g. in human-computer interaction), the study of trust in human-AI interaction in a team setting is still a novel field. For this reason, the MULTITTRUST (Multidisciplinary perspectives on Human-AI Team Trust) workshop series was founded. In this paper, we present the main outcomes after three editions. Our contributions are: an overview of the shared language of concepts and definitions; an outline of the main open research challenges; and methodological guidelines for further studies in meaningful human-AI team trust. These three contributions form a foundational roadmap towards a better understanding of trust in human-AI team interactions.
Conference paper
(2025)
-
Carolina Centeio Jorge, Elena Dumitrescu, Catholijn M. Jonker, Razvan Loghin, Sahar Marossi, Elena Uleia, Myrthe L. Tielman
Mutual trust between humans and interactive artificial agents is crucial for effective human-agent teamwork. This involves not only the human appropriately trusting the artificial teammate, but also the artificial teammate assessing the human’s trustworthiness for different tasks (i.e., artificial trust in human partners). Literature indicated that transparency and explainability is generally beneficial for human-agent collaboration. However, communicating artificial trust potentially affects human trust and satisfaction, which impact team dynamics. Towards studying these effects, we developed an artificial trust model and implemented five distinct communication approaches which varied in modality (visual/graphical and/or text), level (communication and/or explanation), and timing (real-time or occasional). We evaluated the effects of the different communication styles through a user study (N=120) in a 2D grid-world Search and Rescue scenario. Our results show that all our artificial trust explanations improved human trust and satisfaction, but the mere graphical communication of it did not. These results are bound to the specific scenario and context in which this study was run and require further exploration. As such, this work presents a first step towards understanding the consequences of communicating and explaining to a human teammate their assessed trustworthiness.
...
Mutual trust between humans and interactive artificial agents is crucial for effective human-agent teamwork. This involves not only the human appropriately trusting the artificial teammate, but also the artificial teammate assessing the human’s trustworthiness for different tasks (i.e., artificial trust in human partners). Literature indicated that transparency and explainability is generally beneficial for human-agent collaboration. However, communicating artificial trust potentially affects human trust and satisfaction, which impact team dynamics. Towards studying these effects, we developed an artificial trust model and implemented five distinct communication approaches which varied in modality (visual/graphical and/or text), level (communication and/or explanation), and timing (real-time or occasional). We evaluated the effects of the different communication styles through a user study (N=120) in a 2D grid-world Search and Rescue scenario. Our results show that all our artificial trust explanations improved human trust and satisfaction, but the mere graphical communication of it did not. These results are bound to the specific scenario and context in which this study was run and require further exploration. As such, this work presents a first step towards understanding the consequences of communicating and explaining to a human teammate their assessed trustworthiness.
Journal article
(2025)
-
Martin Atzmueller, Carolina Centeio Jorge, Cláudio Rebelo de Sá, Behzad M. Heravi, Jenny L. Gibson, Rosaldo J.F. Rossetti
Social interactions are prevalent in our lives. These can be observed, e. g., online using social media, however, also offline specifically using sensors. In such contexts, typically time-stamped interactions are recorded, which can also be inferred from real-time location of humans. Such interaction data can then be modeled as so-called social interaction networks. For their analysis, a variety of different approaches can be applied. A prominent research direction is then the detection of patterns describing specific subgroups with exceptional behavioral characteristics, given some measure of interest. In the standard case of plain graphs modeling the interaction networks, methods for identifying such subgroups mainly focus on structural characteristics of the network and/or the induced subgraph. For attributed social networks, then additional attributive information can be exploited. This paper proposes to focus on the dyadic structure of the attributed social interaction networks, thus enabling a compositional perspective for identifying interesting subgroup patterns. Specifically, we can then analyze spatio-temporal data modeled as attributed social interaction networks for identifying exceptional social behavior. The presented approach adapts local pattern mining using subgroup discovery to the dyadic setting, exploiting attribute information of the spatio-temporal attributed interaction networks. With this, specific characteristics of social interactions are considered, i. e., duration and frequency, for identifying subgroups capturing social behavior that deviates from the norm. For subgroup discovery, we propose according interestingness measures in the form of seven novel quality functions and discuss their properties. In our experimentation, we perform an evaluation demonstrating the efficacy of the presented approach using four real-world datasets on face-to-face interactions in academic conferencing as well as school playground contexts. Our results indicate that the proposed method returns interesting, meaningful, and valid findings and results.
...
Social interactions are prevalent in our lives. These can be observed, e. g., online using social media, however, also offline specifically using sensors. In such contexts, typically time-stamped interactions are recorded, which can also be inferred from real-time location of humans. Such interaction data can then be modeled as so-called social interaction networks. For their analysis, a variety of different approaches can be applied. A prominent research direction is then the detection of patterns describing specific subgroups with exceptional behavioral characteristics, given some measure of interest. In the standard case of plain graphs modeling the interaction networks, methods for identifying such subgroups mainly focus on structural characteristics of the network and/or the induced subgraph. For attributed social networks, then additional attributive information can be exploited. This paper proposes to focus on the dyadic structure of the attributed social interaction networks, thus enabling a compositional perspective for identifying interesting subgroup patterns. Specifically, we can then analyze spatio-temporal data modeled as attributed social interaction networks for identifying exceptional social behavior. The presented approach adapts local pattern mining using subgroup discovery to the dyadic setting, exploiting attribute information of the spatio-temporal attributed interaction networks. With this, specific characteristics of social interactions are considered, i. e., duration and frequency, for identifying subgroups capturing social behavior that deviates from the norm. For subgroup discovery, we propose according interestingness measures in the form of seven novel quality functions and discuss their properties. In our experimentation, we perform an evaluation demonstrating the efficacy of the presented approach using four real-world datasets on face-to-face interactions in academic conferencing as well as school playground contexts. Our results indicate that the proposed method returns interesting, meaningful, and valid findings and results.
Conference paper
(2024)
-
C. Centeio Jorge, Ewart J. de Visser, M.L. Tielman, C.M. Jonker, Lionel P. Robert
As machines' autonomy increases, their capacity to learn and adapt to humans in collaborative scenarios increases too. In particular, machines can use artificial trust (AT) to make decisions, such as task and role allocation/selection. However, the outcome of such decisions and the way these are communicated can affect the human's trust, which in turn affects how the human collaborates too. With the goal of maintaining mutual appropriate trust between the human and the machine in mind, we reflect on the requirements for having an AT-based decision-making model on an artificial teammate. Furthermore, we propose a user study to investigate the role of task-based willingness (e.g. human preferences on tasks) and its communication in AT-based decision-making.
...
As machines' autonomy increases, their capacity to learn and adapt to humans in collaborative scenarios increases too. In particular, machines can use artificial trust (AT) to make decisions, such as task and role allocation/selection. However, the outcome of such decisions and the way these are communicated can affect the human's trust, which in turn affects how the human collaborates too. With the goal of maintaining mutual appropriate trust between the human and the machine in mind, we reflect on the requirements for having an AT-based decision-making model on an artificial teammate. Furthermore, we propose a user study to investigate the role of task-based willingness (e.g. human preferences on tasks) and its communication in AT-based decision-making.
In teams composed of humans, we use trust in others to make decisions, such as what to do next, who to help and who to ask for help. When a team member is artificial, they should also be able to assess whether a human teammate is trustworthy for a certain task. We see trustworthiness as the combination of (1) whether someone will do a task and (2) whether they can do it. With building beliefs in trustworthiness as an ultimate goal, we explore which internal factors (krypta) of the human may play a role (e.g., ability, benevolence, and integrity) in determining trustworthiness, according to existing literature. Furthermore, we investigate which observable metrics (manifesta) an agent may take into account as cues for the human teammate’s krypta in an online 2D grid-world experiment (n = 54). Results suggest that cues of ability, benevolence and integrity influence trustworthiness. However, we observed that trustworthiness is mainly influenced by human’s playing strategy and cost-benefit analysis, which deserves further investigation. This is a first step towards building informed beliefs of human trustworthiness in human-AI teamwork.
...
In teams composed of humans, we use trust in others to make decisions, such as what to do next, who to help and who to ask for help. When a team member is artificial, they should also be able to assess whether a human teammate is trustworthy for a certain task. We see trustworthiness as the combination of (1) whether someone will do a task and (2) whether they can do it. With building beliefs in trustworthiness as an ultimate goal, we explore which internal factors (krypta) of the human may play a role (e.g., ability, benevolence, and integrity) in determining trustworthiness, according to existing literature. Furthermore, we investigate which observable metrics (manifesta) an agent may take into account as cues for the human teammate’s krypta in an online 2D grid-world experiment (n = 54). Results suggest that cues of ability, benevolence and integrity influence trustworthiness. However, we observed that trustworthiness is mainly influenced by human’s playing strategy and cost-benefit analysis, which deserves further investigation. This is a first step towards building informed beliefs of human trustworthiness in human-AI teamwork.
Book chapter
(2024)
-
Carolina Centeio Jorge, Emma van Zoelen, Ruben Verhagen, Siddharth Mehrotra, Catholijn M. Jonker, Myrthe Tielman
As human-machine teams become a more common scenario, we need to ensure mutual trust between humans and machines. More important than having trust, we need all teammates to trust each other appropriately. This means that they should not overtrust or undertrust each other, avoiding risks and inefficiencies, respectively. We usually think of natural trust, that is, humans trusting machines, but we should also consider artificial trust, that is, artificial agents trusting humans. Appropriate artificial trust allows the agents to interpret human behavior and predict their behavior in a certain context. In this chapter, we explore how we can define this context in terms of task and team characteristics. We present a taxonomy that shows how trust is context-dependent. In fact, we propose that no trust model presented in the literature fits all contexts and argue that our taxonomy facilitates the choice of the trust model that better fits a certain context. The taxonomy helps to understand which internal characteristics of the teammate (krypta) are important to consider and how they will show in behavior cues (manifesta). This taxonomy can also be used to help human-machine teams’ researchers in the problem definition and process of experimental design as it allows a detailed characterization of the task and team configuration. Furthermore, we propose a formalization of the belief of trust as context-dependent trustworthiness, and show how beliefs of trust can be used to reach appropriate trust. Our work provides a starting point to implement mutual appropriate trust in human-machine teams.
...
As human-machine teams become a more common scenario, we need to ensure mutual trust between humans and machines. More important than having trust, we need all teammates to trust each other appropriately. This means that they should not overtrust or undertrust each other, avoiding risks and inefficiencies, respectively. We usually think of natural trust, that is, humans trusting machines, but we should also consider artificial trust, that is, artificial agents trusting humans. Appropriate artificial trust allows the agents to interpret human behavior and predict their behavior in a certain context. In this chapter, we explore how we can define this context in terms of task and team characteristics. We present a taxonomy that shows how trust is context-dependent. In fact, we propose that no trust model presented in the literature fits all contexts and argue that our taxonomy facilitates the choice of the trust model that better fits a certain context. The taxonomy helps to understand which internal characteristics of the teammate (krypta) are important to consider and how they will show in behavior cues (manifesta). This taxonomy can also be used to help human-machine teams’ researchers in the problem definition and process of experimental design as it allows a detailed characterization of the task and team configuration. Furthermore, we propose a formalization of the belief of trust as context-dependent trustworthiness, and show how beliefs of trust can be used to reach appropriate trust. Our work provides a starting point to implement mutual appropriate trust in human-machine teams.
Interdependence and trust analysis (ITA)
A framework for human–machine team design
As machines' autonomy increases, the possibilities for collaboration between a human and a machine also increase. In particular, tasks may be performed with varying levels of interdependence, i.e. from independent to joint actions. The feasibility of each type of interdependence depends on factors that contribute to contextual trustworthiness, such as team members' competence, willingness and external factors. In this paper, we present the Interdependence and Trust Analysis (ITA) framework, which is an extension of Coactive Design's Interdependence Analysis framework (Johnson, M., J. M. Bradshaw, P. J. Feltovich, C. M. Jonker, M. Birna Van Riemsdijk, M. Sierhuis. 2014. Coactive Design: Designing Support for Interdependence in Joint Activity. Journal of Human-Robot Interaction 3 (1): 43–69. https://doi.org/10.5898/JHRI.3.1.Johnson). By including information on contextual trustworthiness, ITA can better support the design of human–machine teams, as well as task allocation and selection. Evaluated through expert interviews and a focus group involving a search and rescue scenario, ITA shows potential as a decision-making tool and a communication bridge among human and machine teammates. Our findings emphasise the need to define tasks and roles based on agent characteristics, and imply that decision-making models should align with human-centred objectives. ITA also highlights the trade-off between utility and effort when designing trustworthy systems, suggesting that guided conversations could improve the team design process. Finally, the ITA framework may improve transparency, justification, and interpretability in decision-making, contributing to appropriate trust among teammates.
...
As machines' autonomy increases, the possibilities for collaboration between a human and a machine also increase. In particular, tasks may be performed with varying levels of interdependence, i.e. from independent to joint actions. The feasibility of each type of interdependence depends on factors that contribute to contextual trustworthiness, such as team members' competence, willingness and external factors. In this paper, we present the Interdependence and Trust Analysis (ITA) framework, which is an extension of Coactive Design's Interdependence Analysis framework (Johnson, M., J. M. Bradshaw, P. J. Feltovich, C. M. Jonker, M. Birna Van Riemsdijk, M. Sierhuis. 2014. Coactive Design: Designing Support for Interdependence in Joint Activity. Journal of Human-Robot Interaction 3 (1): 43–69. https://doi.org/10.5898/JHRI.3.1.Johnson). By including information on contextual trustworthiness, ITA can better support the design of human–machine teams, as well as task allocation and selection. Evaluated through expert interviews and a focus group involving a search and rescue scenario, ITA shows potential as a decision-making tool and a communication bridge among human and machine teammates. Our findings emphasise the need to define tasks and roles based on agent characteristics, and imply that decision-making models should align with human-centred objectives. ITA also highlights the trade-off between utility and effort when designing trustworthy systems, suggesting that guided conversations could improve the team design process. Finally, the ITA framework may improve transparency, justification, and interpretability in decision-making, contributing to appropriate trust among teammates.
Journal article
(2024)
-
Siddharth Mehrotra, Carolina Centeio Jorge, Catholijn M. Jonker, Myrthe L. Tielman
Appropriate trust is an important component of the interaction between people and AI systems, in that "inappropriate"trust can cause disuse, misuse, or abuse of AI. To foster appropriate trust in AI, we need to understand how AI systems can elicit appropriate levels of trust from their users. Out of the aspects that influence trust, this article focuses on the effect of showing integrity. In particular, this article presents a study of how different integrity-based explanations made by an AI agent affect the appropriateness of trust of a human in that agent. To explore this, (1) we provide a formal definition to measure appropriate trust, (2) present a between-subject user study with 160 participants who collaborated with an AI agent in such a task. In the study, the AI agent assisted its human partner in estimating calories on a food plate by expressing its integrity through explanations focusing on either honesty, transparency, or fairness. Our results show that (a) an agent who displays its integrity by being explicit about potential biases in data or algorithms achieved appropriate trust more often compared to being honest about capability or transparent about the decision-making process, and (b) subjective trust builds up and recovers better with honesty-like integrity explanations. Our results contribute to the design of agent-based AI systems that guide humans to appropriately trust them, a formal method to measure appropriate trust, and how to support humans in calibrating their trust in AI.
...
Appropriate trust is an important component of the interaction between people and AI systems, in that "inappropriate"trust can cause disuse, misuse, or abuse of AI. To foster appropriate trust in AI, we need to understand how AI systems can elicit appropriate levels of trust from their users. Out of the aspects that influence trust, this article focuses on the effect of showing integrity. In particular, this article presents a study of how different integrity-based explanations made by an AI agent affect the appropriateness of trust of a human in that agent. To explore this, (1) we provide a formal definition to measure appropriate trust, (2) present a between-subject user study with 160 participants who collaborated with an AI agent in such a task. In the study, the AI agent assisted its human partner in estimating calories on a food plate by expressing its integrity through explanations focusing on either honesty, transparency, or fairness. Our results show that (a) an agent who displays its integrity by being explicit about potential biases in data or algorithms achieved appropriate trust more often compared to being honest about capability or transparent about the decision-making process, and (b) subjective trust builds up and recovers better with honesty-like integrity explanations. Our results contribute to the design of agent-based AI systems that guide humans to appropriately trust them, a formal method to measure appropriate trust, and how to support humans in calibrating their trust in AI.
Establishing an appropriate level of trust between people and AI systems is crucial to avoid the misuse, disuse, or abuse of AI. Understanding how AI systems can generate appropriate levels of trust among users is necessary to achieve this goal. This study focuses on the impact of displaying integrity, which is one of the factors that influence trust. The study analyzes how different integrity-based explanations provided by an AI agent affect a human’s appropriate level of trust in the agent. To explore this, we conducted a between-subject user study involving 160 participants who collaborated with an AI agent to estimate calories on a food plate, with the AI agent expressing its integrity in different ways through explanations. The preliminary results demonstrate that an AI agent that explicitly acknowledges honesty in its decision making process elicit higher subjective trust than those that are transparent about their decision-making process or fair about biases. These findings can aid in designing agent-based AI systems that foster appropriate trust from humans.
...
Establishing an appropriate level of trust between people and AI systems is crucial to avoid the misuse, disuse, or abuse of AI. Understanding how AI systems can generate appropriate levels of trust among users is necessary to achieve this goal. This study focuses on the impact of displaying integrity, which is one of the factors that influence trust. The study analyzes how different integrity-based explanations provided by an AI agent affect a human’s appropriate level of trust in the agent. To explore this, we conducted a between-subject user study involving 160 participants who collaborated with an AI agent to estimate calories on a food plate, with the AI agent expressing its integrity in different ways through explanations. The preliminary results demonstrate that an AI agent that explicitly acknowledges honesty in its decision making process elicit higher subjective trust than those that are transparent about their decision-making process or fair about biases. These findings can aid in designing agent-based AI systems that foster appropriate trust from humans.
Artificial Trust for Decision-Making in Human-AI Teamwork
Steps and Challenges
Human-AI teams count on both humans and artificial agents to work together collaboratively. In human-human teams, we use trust to make decisions. Similarly, our work explores how an AI can use trust (in human teammates) to make decisions while ensuring the team’s goal and mitigating risks for the humans involved. We present the several steps and challenges towards the development of an artificial-trust-based decision-making model.
...
Human-AI teams count on both humans and artificial agents to work together collaboratively. In human-human teams, we use trust to make decisions. Similarly, our work explores how an AI can use trust (in human teammates) to make decisions while ensuring the team’s goal and mitigating risks for the humans involved. We present the several steps and challenges towards the development of an artificial-trust-based decision-making model.
This preface summarises the first Workshop on Multidisciplinary Perspectives on Human-AI Team Trust (MULTITTRUST 2023), co-located with 2nd International Conference on Hybrid Human-Artificial Intelligence (HHAI 2023), held on June 26th 2023 in Munich, Germany.
...
This preface summarises the first Workshop on Multidisciplinary Perspectives on Human-AI Team Trust (MULTITTRUST 2023), co-located with 2nd International Conference on Hybrid Human-Artificial Intelligence (HHAI 2023), held on June 26th 2023 in Munich, Germany.
MULTITTRUST
2nd Workshop on Multidisciplinary Perspectives on Human-AI Team Trust
Conference paper
(2023)
-
Nicolo' Brandizzi, Carolina Centeio Jorge, Roberto Cipollone, Francesco Frattolillo, Luca Iocchi, Anna Sophie Ulfert-Blank
Journal article
(2023)
-
Carolina Centeio Jorge, Nikki H. Bouman, Catholijn M. Jonker, Myrthe L. Tielman
Introduction: Collaboration in teams composed of both humans and automation has an interdependent nature, which demands calibrated trust among all the team members. For building suitable autonomous teammates, we need to study how trust and trustworthiness function in such teams. In particular, automation occasionally fails to do its job, which leads to a decrease in a human’s trust. Research has found interesting effects of such a reduction of trust on the human’s trustworthiness, i.e., human characteristics that make them more or less reliable. This paper investigates how automation failure in a human-automation collaborative scenario affects the human’s trust in the automation, as well as a human’s trustworthiness towards the automation.Methods: We present a 2 × 2 mixed design experiment in which the participants perform a simulated task in a 2D grid-world, collaborating with an automation in a “moving-out” scenario. During the experiment, we measure the participants’ trustworthiness, trust, and liking regarding the automation, both subjectively and objectively.Results: Our results show that automation failure negatively affects the human’s trustworthiness, as well as their trust in and liking of the automation.Discussion: Learning the effects of automation failure in trust and trustworthiness can contribute to a better understanding of the nature and dynamics of trust in these teams and improving human-automation teamwork.
...
Introduction: Collaboration in teams composed of both humans and automation has an interdependent nature, which demands calibrated trust among all the team members. For building suitable autonomous teammates, we need to study how trust and trustworthiness function in such teams. In particular, automation occasionally fails to do its job, which leads to a decrease in a human’s trust. Research has found interesting effects of such a reduction of trust on the human’s trustworthiness, i.e., human characteristics that make them more or less reliable. This paper investigates how automation failure in a human-automation collaborative scenario affects the human’s trust in the automation, as well as a human’s trustworthiness towards the automation.Methods: We present a 2 × 2 mixed design experiment in which the participants perform a simulated task in a 2D grid-world, collaborating with an automation in a “moving-out” scenario. During the experiment, we measure the participants’ trustworthiness, trust, and liking regarding the automation, both subjectively and objectively.Results: Our results show that automation failure negatively affects the human’s trustworthiness, as well as their trust in and liking of the automation.Discussion: Learning the effects of automation failure in trust and trustworthiness can contribute to a better understanding of the nature and dynamics of trust in these teams and improving human-automation teamwork.
Journal article
(2023)
-
Anna-Sophie Ulfert, Eleni Georganta, Carolina Centeio Jorge, Siddharth Mehrotra, Myrthe Tielman
Intelligent systems are increasingly entering the workplace, gradually moving away from technologies supporting work processes to artificially intelligent (AI) agents becoming team members. Therefore, a deep understanding of effective human-AI collaboration within the team context is required. Both psychology and computer science literature emphasize the importance of trust when humans interact either with human team members or AI agents. However, empirical work and theoretical models that combine these research fields and define team trust in human-AI teams are scarce. Furthermore, they often lack to integrate central aspects, such as the multilevel nature of team trust and the role of AI agents as team members. Building on an integration of current literature on trust in human-AI teaming across different research fields, we propose a multidisciplinary framework of team trust in human-AI teams. The framework highlights different trust relationships that exist within human-AI teams and acknowledges the multilevel nature of team trust. We discuss the framework’s potential for human-AI teaming research and for the design and implementation of trustworthy AI team members.
...
Intelligent systems are increasingly entering the workplace, gradually moving away from technologies supporting work processes to artificially intelligent (AI) agents becoming team members. Therefore, a deep understanding of effective human-AI collaboration within the team context is required. Both psychology and computer science literature emphasize the importance of trust when humans interact either with human team members or AI agents. However, empirical work and theoretical models that combine these research fields and define team trust in human-AI teams are scarce. Furthermore, they often lack to integrate central aspects, such as the multilevel nature of team trust and the role of AI agents as team members. Building on an integration of current literature on trust in human-AI teaming across different research fields, we propose a multidisciplinary framework of team trust in human-AI teams. The framework highlights different trust relationships that exist within human-AI teams and acknowledges the multilevel nature of team trust. We discuss the framework’s potential for human-AI teaming research and for the design and implementation of trustworthy AI team members.
Assessing artificial trust in human-agent teams
A conceptual model
As intelligent agents are becoming human's teammates, not only do humans need to trust intelligent agents, but an intelligent agent should also be able to form artificial trust, i.e. a belief regarding human's trustworthiness. We see artificial trust as the beliefs of competence and willingness, and we study which internal factors (krypta) of the human may play a role when assessing artificial trust. Furthermore, we investigate which observable measures (manifesta) an agent may take into account as cues for the human teammate's krypta. This paper proposes a conceptual model of artificial trust for a specific task during human-agent teamwork. Our model proposes observable measures related to human trustworthiness (ability, benevolence, integrity) and strategy (perceived cost and benefit) as predictors for willingness and competence, based on literature and a preliminary user study.
...
As intelligent agents are becoming human's teammates, not only do humans need to trust intelligent agents, but an intelligent agent should also be able to form artificial trust, i.e. a belief regarding human's trustworthiness. We see artificial trust as the beliefs of competence and willingness, and we study which internal factors (krypta) of the human may play a role when assessing artificial trust. Furthermore, we investigate which observable measures (manifesta) an agent may take into account as cues for the human teammate's krypta. This paper proposes a conceptual model of artificial trust for a specific task during human-agent teamwork. Our model proposes observable measures related to human trustworthiness (ability, benevolence, integrity) and strategy (perceived cost and benefit) as predictors for willingness and competence, based on literature and a preliminary user study.
Conference paper
(2022)
-
Jan-Willem van Rhenen, Carolina Centeio Jorge, Tiffany Matej Hrkalovic, Bernd Dudzik
In competitive multiplayer online video games, teamwork is of utmost importance, implying high levels of interdependence between the joint outcomes of players. When engaging in such interdependent interactions, humans rely on trust to facilitate coordination of their individual behaviours. However, online games often take place between teams of strangers, with individual members having little to no information about each other than what they observe throughout the interaction itself. A better understanding of the social behaviours that are used by players to form trust could not only facilitate richer gaming experiences, but could also lead to insights about team interactions. As such, this paper presents a first step towards understanding how and which types of in-game behaviour relate to trust formation. In particular, we investigate a) which in-game behaviour were relevant for trust formation (first part of the study) and b) how they relate to the reported player's trust in their teammates (the second part of the study). The first part consisted of interviews with League of Legends players in order to create a taxonomy of in-game behaviours relevant for trust formation. As for the second part, we ran a small-scale pilot study where participants played the game and then answered a questionnaire to measure their trust in their teammates. Our preliminary results present a taxonomy of in-game behaviours which can be used to annotate the games regarding trust behaviours. Based on the pilot study, the list of behaviours could be extended as to improve the results. These findings can be used to research the role of trust formation in teamwork.
...
In competitive multiplayer online video games, teamwork is of utmost importance, implying high levels of interdependence between the joint outcomes of players. When engaging in such interdependent interactions, humans rely on trust to facilitate coordination of their individual behaviours. However, online games often take place between teams of strangers, with individual members having little to no information about each other than what they observe throughout the interaction itself. A better understanding of the social behaviours that are used by players to form trust could not only facilitate richer gaming experiences, but could also lead to insights about team interactions. As such, this paper presents a first step towards understanding how and which types of in-game behaviour relate to trust formation. In particular, we investigate a) which in-game behaviour were relevant for trust formation (first part of the study) and b) how they relate to the reported player's trust in their teammates (the second part of the study). The first part consisted of interviews with League of Legends players in order to create a taxonomy of in-game behaviours relevant for trust formation. As for the second part, we ran a small-scale pilot study where participants played the game and then answered a questionnaire to measure their trust in their teammates. Our preliminary results present a taxonomy of in-game behaviours which can be used to annotate the games regarding trust behaviours. Based on the pilot study, the list of behaviours could be extended as to improve the results. These findings can be used to research the role of trust formation in teamwork.
Mutual trust is considered a required coordinating mechanism for achieving effective teamwork in human teams. However, it is still a challenge to implement such mechanisms in teams composed by both humans and AI (human-AI teams), even though those are becoming increasingly prevalent. Agents in such teams should not only be trustworthy and promote appropriate trust from the humans, but also know when to trust a human teammate to perform a certain task. In this project, we study trust as a tool for artificial agents to achieve better team work. In particular, we want to build mental models of humans so that agents can understand human trustworthiness in the context of human-AI teamwork, taking into account factors such as human teammates', task's and environment's characteristics.
...
Mutual trust is considered a required coordinating mechanism for achieving effective teamwork in human teams. However, it is still a challenge to implement such mechanisms in teams composed by both humans and AI (human-AI teams), even though those are becoming increasingly prevalent. Agents in such teams should not only be trustworthy and promote appropriate trust from the humans, but also know when to trust a human teammate to perform a certain task. In this project, we study trust as a tool for artificial agents to achieve better team work. In particular, we want to build mental models of humans so that agents can understand human trustworthiness in the context of human-AI teamwork, taking into account factors such as human teammates', task's and environment's characteristics.