The integration of AI systems in medical care magnifies questions related to how physicians should work with such systems to ensure the best patient outcomes. A particularly thorny issue is related to dealing with situations of possible disagreement between an AI system’s recommendation and the course of medical action envisaged by a human clinician. The current academic debate has so far suggested three possible ways of dealing with such clinician-AI disagreements. First, by considering when clinicians are justified in deferring to the AI output (what we call the deference approach), second when the human user overrules the AI system’s output in cases of disagreement (the overruling approach), and lastly when a second human opinion is deemed necessary to resolve disagreements (the second opinion approach). In this paper, we aim to spell out the shortcomings of these three approaches for dealing with clinician-AI disagreement and offer a more nuanced perspective on such disagreements. We argue that differentiation between types of disagreements, taking into account the role attributed to AI in medical practice, is essential before determining how clinician-AI disagreements should be dealt with. By drawing on a case that exemplifies how multifaceted medical decision-making is, we point out the normative implications of possible clinician-AI disagreements ensuing from it. We highlight the distinctive uncertainties inherent to medical decision-making, showing that disagreements in these contexts are not merely unavoidable but can even be epistemically valuable. Ultimately, by considering the epistemic positions of clinicians and AI systems, our analysis raises important questions for the epistemology of disagreement that need timely attention.

AI technologies are increasingly deployed in medical care and decision-making, and efforts geared toward conceptualizing how human control over AI systems can be meaningful, i.e., sufficient to preserve the relevant human agency and responsibility, are mounting. However, a suitable conceptualization of Meaningful Human Control (MHC) explicitly tailored to AI-mediated clinical practice is still underdeveloped. This paper addresses this research gap in two ways. First, it applies the framework of Meaningful Human Control as reason-responsiveness to the medical field. Second, it shows that considerations of epistemic (in)justice ought to be included in efforts toward securing MHC in medical care. MHC demands that the moral reasons of relevant agents be made available to the socio-technical system in which the AI operates. However, this requirement can be compromised by epistemic injustices, i.e., when patients’ and clinicians’ epistemic offerings to the medical discourse are unduly limited. The paper argues that epistemic justice is an important enabler for MHC, and, when properly understood, MHC is a crucial element in a strategy to promote a more just medical AI. Since epistemic injustice depends on power asymmetries and systemic inequalities, achieving epistemic justice and MHC over medical AI requires addressing power and justice issues in the development and use of (new) medical AI.

The increased introduction and use of AI systems in medicine and healthcare span various domains and contexts of applications. While the use of these systems is often met with optimism connected to their opportunity to improve healthcare delivery and optimize cost-efficiency and accuracy (Davenport & Kalakota, 2019), introducing these technologies in health environments is accompanied by concerns at the intersection between ethics and epistemology.

The advances in machine learning (ML)-based systems in medicine give rise to pressing epistemological and ethical questions. Clinical decisions are increasingly taken in highly digitised work environments, which we call artificial epistemic niches. By considering the case of ML systems in life-critical healthcare settings, we investigate (1) when users’ reliance on these systems can be characterised as epistemic dependence and (2) how this dependence turns into what we refer to as harmful epistemic dependence of clinical professionals on medical ML. The latter occurs when the impossibility of critically assessing the soundness of a system’s output in situ implies a moral obligation to comply with its recommendation since a failure to do so constitutes a moral risk that cannot be justified then and there. We analyse the epistemic and moral consequences of harmful epistemic dependence on the status of medical professionals. We conclude by assessing how a suitable design of the epistemic niche can address the problem.

Objective
As hospitals increasingly adopt artificial intelligence (AI) to manage rising patient volumes, workforce shortages and healthcare costs, concerns about ethical implementation have become prominent. This systematic review aims to assess how hospital-focused AI literature addresses the WHO’s six ethical AI principles—autonomy; well-being and safety; transparency and explainability; responsibility and accountability; inclusiveness and equity; and responsiveness and sustainability.

Methods and analysis
A systematic review (PROSPERO registration: CRD42022347871) was conducted by searching Embase, MEDLINE ALL, Web of Science and the Cochrane Central Register of Controlled Trials from inception to December 2023, supplemented by Google Scholar. English-language studies describing AI (machine learning, deep learning, predictive analytics) relevant to inpatient settings and referencing at least one WHO principle were included. Two reviewers independently screened titles, abstracts and full texts, extracting data on publication year, country, study design, AI type, technology readiness level and ethical considerations. Discrepancies were resolved by consensus.

Results
Of 4770 unique records, 673 were included. Most (83%) originated from high-income countries, with publication volume rising sharply after 2021. Of these, 558 (83%) addressed at least one WHO principle in depth, most frequently inclusiveness and equity (49%), transparency and explainability (45%) and autonomy (42%). Well-being and safety (26%) and responsibility and accountability (29%) were less frequently covered, while responsiveness and sustainability (6%) was rarely explored. Among 44 studies developing AI applications with technology readiness levels 1–6, ethical principles were acknowledged but rarely operationalised.

Conclusion
Hospital-based AI research demonstrates increasing attention to ethical principles but lacks comprehensive application, particularly regarding sustainability. High-income countries dominate this discourse, underscoring the need for broader global engagement. To achieve equitable, safe and sustainable AI in clinical practice, clearer operational guidance and more inclusive collaboration is warranted.

Achieving trustworthy AI is increasingly considered an essential desideratum to integrate AI systems into sensitive societal fields, such as criminal justice, finance, medicine, and healthcare, among others. For this reason, it is important to spell out clearly its characteristics, merits, and shortcomings. This article is the first survey in the specialized literature that maps out the philosophical landscape surrounding trust and trustworthiness in AI. To achieve our goals, we proceed as follows. We start by discussing philosophical positions on trust and trustworthiness, focusing on interpersonal accounts of trust. This allows us to explain why trust, in its most general terms, is to be understood as reliance plus some “extra factor”. We then turn to the first part of the definition provided, i.e., reliance, and analyze two opposing approaches to establishing AI systems’ reliability. On the one hand, we consider transparency and, on the other, computational reliabilism. Subsequently, we focus on debates revolving around the “extra factor”. To this end, we consider viewpoints that most actively resist the possibility and desirability of trusting AI systems before turning to the analysis of the most prominent advocates of it. Finally, we take up the main conclusions of the previous sections and briefly point at issues that remain open and need further attention.

Considering the overall shortage of therapists to meet the psychological needs of vulnerable populations, AI-based technologies are often seen as a possible remedy. Particularly smartphone apps or chatbots are increasingly used to offer mental health support, mostly through cognitive behavioral therapy. The assumption underlying the deployment of these systems is their ability to make mental health support accessible to generally underserved populations. Hence, this seems to be aligned with the fundamental biomedical principle of justice understood in its distributive meaning. However, considerations of the principle of justice in its epistemic significance are still in their infancy in the debates revolving around the ethical issues connected to the use of mental health chatbots. This paper aims to fill this research gap, focusing on a less familiar kind of harm that these systems can cause, namely the harm to users in their capacities as knowing subjects. More specifically, we frame our discussion in terms of one form of epistemic injustice that such practices are especially prone to bring about, i.e., participatory injustice. To make our theoretical analysis more graspable and to show its urgency, we discuss the case of a mental health Chatbot, Karim, deployed to deliver mental health support to Syrian refugees. This case substantiates our theoretical considerations and the epistemo-ethical concerns arising from the use of mental health applications among vulnerable populations. Finally, we argue that conceptualizing epistemic participation as a capability within the framework of Capability Sensitive Design can be a first step toward ameliorating the participatory injustice discussed in this paper.

The advancement of AI-based technologies, such as machine learning (ML) systems, for implementation in healthcare is progressing rapidly. Since these systems are used to support healthcare professionals in crucial medical practices, their role in medical decision-making needs to be epistemologically and ethically assessed. However, a central issue at the intersection of the ethics and epistemology of ML has been largely neglected. This pertains to the careful scrutiny of how ML systems can degrade individuals’ epistemic standing as receivers and conveyors of knowledge and, thereby, perpetrate epistemic injustice. Since ML systems are powerful epistemic entities that are not easily contestable, and their decision-making rationale is often inaccessible, it is crucial to consider their role in creating imbalances in patients’ disfavor and the ways to mitigate such imbalances. This is especially important when it comes to interactions between patients and physicians, in which questions of credibility, trust, and understanding are central. Against this background, the overarching purpose of this dissertation is to fill this research gap by providing a framework to identify and, on occasion, mitigate epistemic injustices that are ML-induced, i.e., that emerge specifically due to the role that ML systems play in patient-physician interactions.

The principle of trust has been placed at the centre as an attitude for engaging with clinical machine learning systems. However, the notions of trust and distrust remain fiercely debated in the philosophical and ethical literature. In this article, we proceed on a structural level ex negativo as we aim to analyse the concept of “institutional distrustworthiness” to achieve a proper diagnosis of how we should not engage with medical machine learning. First, we begin with several examples that hint at the emergence of a climate of distrust in the context of medical machine learning. Second, we introduce the concept of institutional trustworthiness based on an expansion of Hawley’s commitment account. Third, we argue that institutional opacity can undermine the trustworthiness of medical institutions and can lead to new forms of testimonial injustices. Finally, we focus on possible building blocks for repairing institutional distrustworthiness.

In this paper, we discuss epistemic and ethical concerns brought about by machine learning (ML) systems implemented in medicine. We begin by fleshing out the logic underlying a common approach in the specialized literature (which we call the informativeness account). We maintain that the informativeness account limits its analysis to the impact of epistemological issues on ethical concerns without assessing the bearings that ethical features have on the epistemological evaluation of ML systems. We argue that according to this methodological approach, epistemological issues are instrumental to and autonomous of ethical considerations. This means that the informativeness account considers epistemological evaluation uninfluenced and unregulated by an ethical counterpart. Using an example that does not square well into the informativeness account, we argue for ethical assessments that have a substantial influence on the epistemological assessment of ML and that such influence should not be understood as merely informative but rather regulatory. Drawing on the case analyzed, we claim that within the theoretical framework of the informativeness approach, forms of epistemic injustice—especially epistemic objectification—remain unaddressed. Our analysis should motivate further research investigating the regulatory role that ethical elements play in the epistemology of ML.

In their contribution, Ugar and Malele shed light on an often overlooked but crucial aspect of the ethical development of machine learning (ML) systems to support the diagnosis of mental health disorders. The authors restrain their focus on pointing to the danger of misdiagnosing mental health pathologies that do not qualify as such within sub-Saharan African communities and argue for the need to include population-specific values in these technologies’ design. However, an analysis of the nature of the harm caused to said populations once their values remain unrecognised is not offered. [...]

The social aspects of causality in medicine and healthcare have been emphasized in recent debates in the philosophy of science as crucial factors that need to be considered to enable, among others, appropriate interventions in public health. Therefore, it seems central to recognize the bearing of social causes (broadly understood, e.g., social inequalities and socio-economic status) in bringing about certain concrete pathologies. Being aware of the relevance of social causes in medicine and healthcare is particularly important in the face of the role that artificial intelligence-based systems (such as machine learning algorithms) are increasingly playing in these high-stakes fields. In fact, these systems bear the dangerous potential of concealing relevant social causes. This is highly problematic not only because it reinforces issues of distributive injustice but also because it can pave the way for issues of epistemic injustice. The central aim of this chapter is to make a first effort to point out possible connections between the importance of recognizing social causes in medicine and healthcare and forms of epistemic injustice.

Artificial intelligence-based (AI) technologies such as machine learning (ML) systems are playing an increasingly relevant role in medicine and healthcare, bringing about novel ethical and epistemological issues that need to be timely addressed. Even though ethical questions connected to epistemic concerns have been at the center of the debate, it is going unnoticed how epistemic forms of injustice can be ML-induced, specifically in healthcare. I analyze the shortcomings of an ML system currently deployed in the USA to predict patients’ likelihood of opioid addiction and misuse (PDMP algorithmic platforms). Drawing on this analysis, I aim to show that the wrong inflicted on epistemic agents involved in and affected by these systems’ decision-making processes can be captured through the lenses of Miranda Fricker’s account of hermeneutical injustice. I further argue that ML-induced hermeneutical injustice is particularly harmful due to what I define as an automated hermeneutical appropriation from the side of the ML system. The latter occurs if the ML system establishes meanings and shared hermeneutical resources without allowing for human oversight, impairing understanding and communication practices among stakeholders involved in medical decision-making. Furthermore and very much crucially, an automated hermeneutical appropriation can be recognized if physicians are strongly limited in their possibilities to safeguard patients from ML-induced hermeneutical injustice. Overall, my paper should expand the analysis of ethical issues raised by ML systems that are to be considered epistemic in nature, thus contributing to bridging the gap between these two dimensions in the ongoing debate.

Machine learning (ML) systems play an increasingly relevant role in medicine and healthcare. As their applications move ever closer to patient care and cure in clinical settings, ethical concerns about the responsibility of their use come to the fore. I analyse an aspect of responsible ML use that bears not only an ethical but also a significant epistemic dimension. I focus on ML systems' role in mediating patient-physician relations. I thereby consider how ML systems may silence patients' voices and relativise the credibility of their opinions, which undermines their overall credibility status without valid moral and epistemic justification. More specifically, I argue that withholding credibility due to how ML systems operate can be particularly harmful to patients and, apart from adverse outcomes, qualifies as a form of testimonial injustice. I make my case for testimonial injustice in medical ML by considering ML systems currently used in the USA to predict patients' risk of misusing opioids (automated Prediction Drug Monitoring Programmes, PDMPs for short). I argue that the locus of testimonial injustice in ML-mediated medical encounters is found in the fact that these systems are treated as markers of trustworthiness on which patients' credibility is assessed. I further show how ML-based PDMPs exacerbate and further propagate social inequalities at the expense of vulnerable social groups.

In my paper entitled 'Testimonial injustice in medical machine learning',1 I argued that machine learning (ML)-based Prediction Drug Monitoring Programmes (PDMPs) could infringe on patients' epistemic and moral standing inflicting a testimonial injustice.2 I am very grateful for all the comments the paper received, some of which expand on it while others take a more critical view. This response addresses two objections raised to my consideration of ML-induced testimonial injustice in order to clarify the position taken in the paper. The first maintains that my critical stance toward ML-based PDMPs idealises standard medical practice. Moreover, it claims that the ML-induced testimonial injustice I discuss is not substantially different from situations in which it emerges in human-human interactions. The second claims that my analysis does not establish a link to issues of automation bias, even if these are to be considered the core of testimonial injustice in ML.