Appropriate Trust in Artificial Intelligence (AI) systems has rapidly become an important area of focus for both researchers and practitioners. Various approaches have been used to achieve it, such as confidence scores, explanations, trustworthiness cues, or uncertainty communication. However, a comprehensive understanding of the field is lacking due to the diversity of perspectives arising from various backgrounds that influence it and the lack of a single definition for appropriate trust. To investigate this topic, this paper presents a systematic review to identify current practices in building appropriate trust, different ways to measure it, types of tasks used, and potential challenges associated with it. We also propose a Belief, Intentions, and Actions (BIA) mapping to study commonalities and differences in the concepts related to appropriate trust by (a) describing the existing disagreements on defining appropriate trust, and (b) providing an overview of the concepts and definitions related to appropriate trust in AI from the existing literature. Finally, the challenges identified in studying appropriate trust are discussed, and observations are summarized as current trends, potential gaps, and research opportunities for future work. Overall, the paper provides insights into the complex concept of appropriate trust in human-AI interaction and presents research opportunities to advance our understanding on this topic.@en

Purpose The objective of this paper is to develop a redesigned commissioning process for social care services that fosters integrated care, encourages collaboration and balances professional expertise with client engagement. Design/methodology/approach This study employs a two-pronged approach: a case study of a municipality’s use of subsidy tables and a literature scoping review on integrated care research. Findings The paper introduces a new framework for the study of the new “subsidy tables.” A well-defined and extensive consultation process involving both social care providers (suppliers), the Service Triad, and client representation adds to the existing research on supplier consultation, and on how to define the outcomes for clients via client engagement. Research limitations/implications While aspects are clearly relevant to the Netherlands, the design of the commissioning process of social care has international relevance as well: finding definitions, formulating outcomes and incentives, designing a more collaborative instead of competitive process, stakeholder engagement and consultation. Practical implications Several Dutch municipalities started using the “subsidy tables” method for commissioning integrated social care. This paper offers clear improvements that benefit the commissioners, the social care providers and their clients. Social implications Improving the commissioning process of integrated social care will lead to better fitting care for people who need social care. Originality/value This paper is one of the first to do a thorough analysis of the “subsidy tables” method for commissioning integrated social care. @en

As human-machine teams become a more common scenario, we need to ensure mutual trust between humans and machines. More important than having trust, we need all teammates to trust each other appropriately. This means that they should not overtrust or undertrust each other, avoiding risks and inefficiencies, respectively. We usually think of natural trust, that is, humans trusting machines, but we should also consider artificial trust, that is, artificial agents trusting humans. Appropriate artificial trust allows the agents to interpret human behavior and predict their behavior in a certain context. In this chapter, we explore how we can define this context in terms of task and team characteristics. We present a taxonomy that shows how trust is context-dependent. In fact, we propose that no trust model presented in the literature fits all contexts and argue that our taxonomy facilitates the choice of the trust model that better fits a certain context. The taxonomy helps to understand which internal characteristics of the teammate (krypta) are important to consider and how they will show in behavior cues (manifesta). This taxonomy can also be used to help human-machine teams’ researchers in the problem definition and process of experimental design as it allows a detailed characterization of the task and team configuration. Furthermore, we propose a formalization of the belief of trust as context-dependent trustworthiness, and show how beliefs of trust can be used to reach appropriate trust. Our work provides a starting point to implement mutual appropriate trust in human-machine teams.@en

Large-scale survey tools enable the collection of citizen feedback in opinion corpora. Extracting the key arguments from a large and noisy set of opinions helps in understanding the opinions quickly and accurately. Fully automated methods can extract arguments but (1) require large labeled datasets that induce large annotation costs and (2) work well for known viewpoints, but not for novel points of view. We propose HyEnA, a hybrid (human + AI) method for extracting arguments from opinionated texts, combining the speed of automated processing with the understanding and reasoning capabilities of humans. We evaluate HyEnA on three citizen feedback corpora. We find that, on the one hand, HyEnA achieves higher coverage and precision than a state-of-The-Art automated method when compared to a common set of diverse opinions, justifying the need for human insight. On the other hand, HyEnA requires less human effort and does not compromise quality compared to (fully manual) expert analysis, demonstrating the benefit of combining human and artificial intelligence.

@en

We adopt an emerging and prominent vision of human-centred Artificial Intelligence that requires building trustworthy intelligent systems. Such systems should be capable of dealing with the challenges of an interconnected, globalised world by handling plurality and by abiding by human values. Within this vision, pluralistic value alignment is a core problem for AI– that is, the challenge of creating AI systems that align with a set of diverse individual value systems. So far, most literature on value alignment has considered alignment to a single value system. To address this research gap, we propose a novel method for estimating and aggregating multiple individual value systems. We rely on recent results in the social choice literature and formalise the value system aggregation problem as an optimisation problem. We then cast this problem as an ℓ_p-regression problem. Doing so provides a principled and general theoretical framework to model and solve the aggregation problem. Our aggregation method allows us to consider a range of ethical principles, from utilitarian (maximum utility) to egalitarian (maximum fairness). We illustrate the aggregation of value systems by considering real-world data from two case studies: the Participatory Value Evaluation process and the European Values Study. Our experimental evaluation shows how different consensus value systems can be obtained depending on the ethical principle of choice, leading to practical insights for a decision-maker on how to perform value system aggregation.

@en

With the growing capabilities and pervasiveness of AI systems, societies must collectively choose between reduced human autonomy, endangered democracies and limited human rights, and AI that is aligned to human and social values, nurturing collaboration, resilience, knowledge and ethical behaviour. In this chapter, we introduce the notion of self-reflective AI systems for meaningful human control over AI systems. Focusing on decision support systems, we propose a framework that integrates knowledge from psychology and philosophy with formal reasoning methods and machine learning approaches to create AI systems responsive to human values and social norms. We also propose a possible research approach to design and develop self-reflective capability in AI systems. Finally, we argue that self-reflective AI systems can lead to self-reflective hybrid systems (human + AI), thus increasing meaningful human control and empowering human moral reasoning by providing comprehensible information and insights on possible human moral blind spots. @en

Purpose: The objective of this paper is to develop a redesigned commissioning process for social care services that fosters integrated care, encourages collaboration and balances professional expertise with client engagement. Design/methodology/approach: This study employs a two-pronged approach: a case study of a municipality’s use of subsidy tables and a literature scoping review on integrated care research. Findings: The paper introduces a new framework for the study of the new “subsidy tables.” A well-defined and extensive consultation process involving both social care providers (suppliers), the Service Triad, and client representation adds to the existing research on supplier consultation, and on how to define the outcomes for clients via client engagement. Research limitations/implications: While aspects are clearly relevant to the Netherlands, the design of the commissioning process of social care has international relevance as well: finding definitions, formulating outcomes and incentives, designing a more collaborative instead of competitive process, stakeholder engagement and consultation. Practical implications: Several Dutch municipalities started using the “subsidy tables” method for commissioning integrated social care. This paper offers clear improvements that benefit the commissioners, the social care providers and their clients. Social implications: Improving the commissioning process of integrated social care will lead to better fitting care for people who need social care. Originality/value: This paper is one of the first to do a thorough analysis of the “subsidy tables” method for commissioning integrated social care.

@en

The complexity of automated negotiation research calls for dedicated, user-friendly research frameworks that facilitate advanced analytics, comprehensive loggers, visualization tools, and auto-generated domains and preference profiles. This paper introduces NegoLog, a platform that provides advanced and customizable analysis modules to agent developers for exhaustive performance evaluation. NegoLog introduces an automated scenario and tournament generation tool in its Web-based user interface so that the agent developers can adjust the competitiveness and complexity of the negotiations. One of the key novelties of the NegoLog is an individual assessment of preference estimation models independent of the strategies.

@en

Epistemic logic can be used to reason about statements such as ‘I know that you know that I know that φ ’. In this logic, and its extensions, it is commonly assumed that agents can reason about epistemic statements of arbitrary nesting depth. In contrast, empirical findings on Theory of Mind, the ability to (recursively) reason about mental states of others, show that human recursive reasoning capability has an upper bound. In the present paper we work towards resolving this disparity by proposing some elements of a logic of bounded Theory of Mind, built on Public Announcement Logic. Using this logic, and a statistical method called Random-Effects Bayesian Model Selection, we estimate the distribution of Theory of Mind levels in the participant population of a previous behavioral experiment. Despite not modeling stochastic behavior, we find that approximately three-quarters of participants’ decisions can be described using Theory of Mind. In contrast to previous empirical research, our models estimate the majority of participants to be second-order Theory of Mind users.

@en

Relevancy is a prevalent term in value alignment. We either need to keep track of the relevant moral reasons, we need to embed the relevant values, or we need to learn from the relevant behaviour. What relevancy entails in particular cases, however, is often ill-defined. The reasons for this are obvious, it is hard to define relevancy in a way that is both general and concrete enough to give direction towards a specific implementation. In this paper, we describe the inherent difficulty that comes along with defining what is relevant to a particular situation. Simply due to design and the way an AI system functions, we need to state or learn particular goals and circumstances under which that goal is completed. However, because of both the changing nature of the world and the varied wielders and users of such implements, misalignment occurs, especially after a longer amount of time. We propose a way to counteract this by putting contestability front and centre throughout the lifecycle of an AI system, as it can provide insight into what is actually relevant at a particular instance. This allows designers to update the applications in such a manner that they can account for oversight during design.@en

Appropriate trust is an important component of the interaction between people and AI systems, in that "inappropriate"trust can cause disuse, misuse, or abuse of AI. To foster appropriate trust in AI, we need to understand how AI systems can elicit appropriate levels of trust from their users. Out of the aspects that influence trust, this article focuses on the effect of showing integrity. In particular, this article presents a study of how different integrity-based explanations made by an AI agent affect the appropriateness of trust of a human in that agent. To explore this, (1) we provide a formal definition to measure appropriate trust, (2) present a between-subject user study with 160 participants who collaborated with an AI agent in such a task. In the study, the AI agent assisted its human partner in estimating calories on a food plate by expressing its integrity through explanations focusing on either honesty, transparency, or fairness. Our results show that (a) an agent who displays its integrity by being explicit about potential biases in data or algorithms achieved appropriate trust more often compared to being honest about capability or transparent about the decision-making process, and (b) subjective trust builds up and recovers better with honesty-like integrity explanations. Our results contribute to the design of agent-based AI systems that guide humans to appropriately trust them, a formal method to measure appropriate trust, and how to support humans in calibrating their trust in AI.

@en

As machines' autonomy increases, the possibilities for collaboration between a human and a machine also increase. In particular, tasks may be performed with varying levels of interdependence, i.e. from independent to joint actions. The feasibility of each type of interdependence depends on factors that contribute to contextual trustworthiness, such as team members' competence, willingness and external factors. In this paper, we present the Interdependence and Trust Analysis (ITA) framework, which is an extension of Coactive Design's Interdependence Analysis framework (Johnson, M., J. M. Bradshaw, P. J. Feltovich, C. M. Jonker, M. Birna Van Riemsdijk, M. Sierhuis. 2014. Coactive Design: Designing Support for Interdependence in Joint Activity. Journal of Human-Robot Interaction 3 (1): 43–69. https://doi.org/10.5898/JHRI.3.1.Johnson). By including information on contextual trustworthiness, ITA can better support the design of human–machine teams, as well as task allocation and selection. Evaluated through expert interviews and a focus group involving a search and rescue scenario, ITA shows potential as a decision-making tool and a communication bridge among human and machine teammates. Our findings emphasise the need to define tasks and roles based on agent characteristics, and imply that decision-making models should align with human-centred objectives. ITA also highlights the trade-off between utility and effort when designing trustworthy systems, suggesting that guided conversations could improve the team design process. Finally, the ITA framework may improve transparency, justification, and interpretability in decision-making, contributing to appropriate trust among teammates.

@en

Disagreements are common in online societal deliberation and may be crucial for effective collaboration, for instance in helping users understand opposing viewpoints. Although there exist automated methods for recognizing disagreement, a deeper understanding of factors that influence disagreement is currently missing. We investigate a hypothesis that differences in personal values influence disagreement in online discussions. Using Large Language Models (LLMs) for estimating both profiles of personal values and disagreement, we conduct a large-scale experiment involving 11.4M user comments. We find that the dissimilarity of value profiles correlates with disagreement only in specific cases, but that incorporating self-reported value profiles changes these results to be more undecided.@en

In teams composed of humans, we use trust in others to make decisions, such as what to do next, who to help and who to ask for help. When a team member is artificial, they should also be able to assess whether a human teammate is trustworthy for a certain task. We see trustworthiness as the combination of (1) whether someone will do a task and (2) whether they can do it. With building beliefs in trustworthiness as an ultimate goal, we explore which internal factors (krypta) of the human may play a role (e.g., ability, benevolence, and integrity) in determining trustworthiness, according to existing literature. Furthermore, we investigate which observable metrics (manifesta) an agent may take into account as cues for the human teammate’s krypta in an online 2D grid-world experiment (n = 54). Results suggest that cues of ability, benevolence and integrity influence trustworthiness. However, we observed that trustworthiness is mainly influenced by human’s playing strategy and cost-benefit analysis, which deserves further investigation. This is a first step towards building informed beliefs of human trustworthiness in human-AI teamwork.@en

Understanding behavior of human drivers in interactions with automated vehicles (AV) can aid the development of future AVs. Existing investigations of such behavior have predominantly focused on situations in which an AV a priori needs to take action because the human has the right of way. However, future AVs might need to proactively manage interactions even if they have the right of way over humans, e.g., a human driver taking a left turn in front of the approaching AV. Yet it remains unclear how AVs could behave in such interactions and how humans would react to them. To address this issue, here we investigated behavior of human drivers (N = 19) when interacting with an oncoming AV during unprotected left turns in a driving simulator experiment. We measured the outcomes (Go or Stay) and timing of participants’ decisions when interacting with an AV which performed subtle longitudinal nudging maneuvers, e.g. briefly decelerating and then accelerating back to its original speed. We found that participants’ behavior was sensitive to deceleration nudges but not acceleration nudges. We compared the obtained data to predictions of several variants of a drift-diffusion model of human decision making. The most parsimonious model that captured the data hypothesized noisy integration of dynamic information on time-to-arrival and distance to a fixed decision boundary, with an initial accumulation bias towards the Go decision. Our model not only accounts for the observed behavior but can also flexibly generate predictions of human responses to arbitrary longitudinal AV maneuvers, and can be used for both informing future studies of human behavior and incorporating insights from such studies into computational frameworks for AV interaction planning.@en

Presenting high-level arguments is a crucial task for fostering participation in online societal discussions. Current argument summarization approaches miss an important facet of this task-capturing diversity-which is important for accommodating multiple perspectives. We introduce three aspects of diversity: those of opinions, annotators, and sources. We evaluate approaches to a popular argument summarization task called Key Point Analysis, which shows how these approaches struggle to (1) represent arguments shared by few people, (2) deal with data from various sources, and (3) align with subjectivity in human-provided annotations. We find that both general-purpose LLMs and dedicated KPA models exhibit this behavior, but have complementary strengths. Further, we observe that diversification of training data may ameliorate generalization. Addressing diversity in argument summarization requires a mix of strategies to deal with subjectivity.

@en

Many technological systems these days interact with their environment with increasingly little human intervention. This situation comes with higher stakes and consequences that society needs to manage. No longer are we dealing with 404 pages: AI systems today may cause serious harm. To address this, we wish to exert a kind of control over these systems, so that they can adhere to our moral beliefs. However, given the plurality of values in our societies, which “oughts” ought these machines to adhere to? In this article, we examine Borda voting as a way to maximize expected choice-worthiness among individuals through different possible “implementations” of ethical principles. We use data from the Moral Machine experiment to illustrate the effectiveness of such a voting system. Although it appears to be effective on average, the maximization of expected choice-worthiness is heavily dependent on the formulation of principles. While Borda voting may be a good way of ensuring outcomes that are preferable to many, the larger problems in maximizing expected choice-worthiness, such as the capacity to formulate credences well, remain notoriously difficult; hence, we argue that such mechanisms should be implemented with caution and that other problems ought to be solved first.@en

This paper presents the negotiation support mechanisms provided by the Pocket Negotiator (PN) and an elaborate empirical evaluation of the economic decision support (EDS) mechanisms during the bidding phase of negotiations as provided by the PN. Some of these support mechanisms are offered actively, some passively. With passive support we mean that the user only gets that support by clicking a button, whereas active support is provided without prompting. Our results show, that PN improves negotiation outcomes, counters cognitive depletion, and encourages exploration of potential outcomes. We found that the active mechanisms were used more effectively than the passive ones and, overall, the various mechanisms were not used optimally, which opens up new avenues for research. As expected, the participants with higher negotiation skills outperformed the other groups, but still they benefited from PN support. Our experimental results show that people with enough technical skills and with some basic negotiation knowledge will benefit most from PN support. Our results also show that the cognitive depletion effect is reduced by Pocket Negotiator support. The questionnaire taken after the experiment shows that overall the participants found Pocket Negotiator easy to interact with, that it made them negotiate more quickly and that it improves their outcome. Based on our findings, we recommend to 1) provide active support mechanisms (push) to nudge users to be more effective, and 2) provide support mechanisms that shield the user from mathematical complexities.

@en

Channel allocation in dense, decentralized Wi-Fi networks is a challenging due to the highly nonlinear solution space and the difficulty to estimate the opponent’s utility model. So far, only centralized or mediated approaches have succeeded in applying negotiation to this setting. We propose the first two fully-distributed negotiation approaches for Wi-Fi channel assignment. Both of them leverage a pre-sampling of the utility space with simulated annealing and a noisy estimation of the Wi-Fi utility function. Regarding negotiation protocols, one of the approaches makes use of the Alternating Offers protocol, while the other uses the novel Multiple Offers Protocol for Multilateral Negotiations with Partial Consensus (MOPaC), which naturally matches the problem peculiarities. We compare the performance of our proposed approaches with the previous mediated approach, based on simple text mediation. Our experiments show that our approaches yield better utility outcomes, better fairness and less information disclosure than the mediated approach.

@en

As artificial agents become increasingly embedded in our society, we must ensure that their behavior aligns with human values. Value alignment entails value inference, the process of identifying values and reasoning about how humans prioritize values. We introduce a holistic framework that connects the technical (AI) components necessary for value inference. Subsequently, we discuss how hybrid intelligence'the synergy of human and artificial intelligence'is instrumental to the success of value inference. Finally, we illustrate how value inference both poses significant challenges and provides novel opportunities for multiagent systems research.@en