This handbook presents the concept of ‘meaningful human control’ (MHC) over AI systems from the perspectives of (i) philosophy and ethics, (ii) law and governance, and (iii) design and engineering. The introductory chapter addresses the motivations and recent developments in MHC, introducing each perspective and related chapters. These three disciplinary perspectives scrutinize how MHC intertwines with philosophical debates on moral responsibility, societal concerns regarding control over technological advancements in legal frameworks, and the engineering complexities of designing and developing AI systems while ensuring human control and responsibility. Additionally, cross-cutting aspects on MHC over AI systems are also introduced and discussed through (iv) interdisciplinary and systemic perspectives. By offering a contextualized introduction to the perspectives considered in this handbook, this chapter aims to present the handbook’s various approaches and points of interest for a diverse audience, highlighting potential entry points into this multidisciplinary volume.

This book claims that artificial intelligence (AI) may affect our freedom at work, in our daily life, and in the political sphere. The author provides a philosophical framework to help make sense of and govern the ethical and political impact of AI in these domains. AI presents great opportunities and risks, raising the question of how to reap its potential benefits without endangering basic human and societal values. The author identifies three major risks for human freedom. First, AI offers employers new forms of control of the workforce, opening the door to new forms of domination and exploitation. Second, it may reduce our capacity to remain in control of and responsible for our decisions and actions, thereby affecting our free will and moral responsibility. Third, it may increase the power of governments and tech companies to steer the political debate, thereby affecting the possibility of a free and inclusive political participation. The author claims that it is still possible to promote human freedom in our interactions with AI. This requires designing AI systems that help promote workers’ freedom, strengthen human control and responsibility, and foster a free, active, and inclusive democratic participation. Human Freedom in the Age of AI will be of interest to scholars and graduate students working on the ethics of technology, philosophy of technology, political philosophy, design, and artificial intelligence.

The urban traffic environment is characterized by the presence of a highly differentiated pool of users, including vulnerable ones. This makes vehicle automation particularly difficult to implement, as a safe coordination among those users is hard to achieve in such an open scenario. Different strategies have been proposed to address these coordination issues, but all of them have been found to be costly for they negatively affect a range of human values (e.g. safety, democracy, accountability…). In this paper, we claim that the negative value impacts entailed by each of these strategies can be interpreted as lack of what we call Meaningful Human Control over different parts of a sociotechnical system. We argue that Meaningful Human Control theory provides the conceptual tools to reduce those unwanted consequences, and show how “designing for meaningful human control” constitutes a valid strategy to address coordination issues. Furthermore, we showcase a possible application of this framework in a highly dynamic urban scenario, aiming to safeguard important values such as safety, democracy, individual autonomy, and accountability. Our meaningful human control framework offers a perspective on coordination issues that allows to keep human actors in control while minimizing the active, operational role of the drivers. This approach makes ultimately possible to promote a safe and responsible transition to full automation.

The present chapter analyses the complex ways in which driving automation affects human autonomy with the aim of raising awareness on the design and policy challenges that must be faced to effectively align future transportation systems to this ethical value. Building on the European report Ethics of Connected and Automated Vehicles, we consider three dimensions of the relation between human autonomy and driving automation: autonomy as self-determination of driving decisions; autonomy as freedom to pursue a good life through mobility; and, finally, autonomy as the capacity and opportunity to influence social deliberation concerning transportation policies and planning. In doing so, the chapter shows that delegating driving tasks to CAVs might both infringe and support user autonomy, thus calling for a reconsideration of widespread frameworks concerning the role of humans and technological systems in this domain. Moreover, it stresses the importance of promoting inclusive and participated decision-making processes on transportation policies and planning, so to avoid situations where the development and adoption of transport innovations are led by agents willing to respond only to a limited set of stakeholders’ needs.

The paper presents a framework to realise “meaningful human control” over Automated Driving Systems. The framework is based on an original synthesis of the results of the multidisciplinary research project “Meaningful Human Control over Automated Driving Systems” lead by a team of engineers, philosophers, and psychologists at Delft University of the Technology from 2017 to 2021. Meaningful human control aims at protecting safety and reducing responsibility gaps. The framework is based on the core assumption that human persons and institutions, not hardware and software and their algorithms, should remain ultimately—though not necessarily directly—in control of, and thus morally responsible for, the potentially dangerous operation of driving in mixed traffic. We propose an Automated Driving System to be under meaningful human control if it behaves according to the relevant reasons of the relevant human actors (tracking), and that any potentially dangerous event can be related to a human actor (tracing). We operationalise the requirements for meaningful human control through multidisciplinary work in philosophy, behavioural psychology and traffic engineering. The tracking condition is operationalised via a proximal scale of reasons and the tracing condition via an evaluation cascade table. We review the implications and requirements for the behaviour and skills of human actors, in particular related to supervisory control and driver education. We show how the evaluation cascade table can be applied in concrete engineering use cases in combination with the definition of core components to expose deficiencies in traceability, thereby avoiding so-called responsibility gaps. Future research directions are proposed to expand the philosophical framework and use cases, supervisory control and driver education, real-world pilots and institutional embedding

Automated vehicles (AVs) aim to dramatically improve traffic safety by reducing or eliminating human error, which remains the leading cause of road crashes. However, commonly accepted standards for the ‘safe driving behaviour of machines’ are pending and urgently needed. Unless a common understanding of safety as a design value is achieved, different manufacturers’ driving styles may emerge, resulting in inconsistent, unpredictable and potentially unsafe ‘behaviour’ of AVs in certain situations. This paper aims to explore the main gaps and challenges towards establishing shared safety standards for the ‘behaviour’ of AVs, and contribute to their responsible traffic integration, by reviewing the state-of-the-art on AV safety in the core relevant disciplines: ethics of technology, safety science (engineering & human factors), and standardisation. The ethical and safety aspects investigated include the users’ perception of AV safety, the ethical trade-offs in critical decision-making contexts, the pertinence of data-driven approaches for AVs to mimic human behaviour, and the responsibilities of various actors. Moreover, the paper reviews the current safety patterns, metrics (surrogate measures of safety – SMoS) and their thresholds introduced in existing research for three use cases: mixed traffic of AV and conventional vehicles, AV interaction with pedestrians and cyclists, and transition of control from machine to human driver. The results reveal several knowledge gaps within each discipline and highlights the lack of common understanding of safety across disciplines. On the basis of the results, the paper proposes a framework for further research on AV safety, identifying concrete opportunities for interdisciplinary research, with common goals and methodologies, and explicitly indicating the path for transfer of knowledge between sectors.

The paper has two goals. The first is presenting the main results of the recent report Ethics of Connected and Automated Vehicles: recommendations on road safety, privacy, fairness, explainability and responsibility written by the Horizon 2020 European Commission Expert Group to advise on specific ethical issues raised by driverless mobility, of which the author of this paper has been member and rapporteur. The second is presenting some broader ethical and philosophical implications of these recommendations, and using these to contribute to the establishment of Ethics of Transportation as an independent branch of applied ethics. The recent debate on the ethics of Connected and Automated Vehicles (CAVs) presents a paradox and an opportunity. The paradox is the presence of a flourishing debate on the ethics of one very specific transportation technology without ethics of transportation being in itself a well-established academic discipline. The opportunity is that now that a spotlight has been switched on the ethical dimensions of CAVs it may be easier to establish a broader debate on ethics of transportation. While the 20 recommendations of the EU report are grouped in three macro-areas: road safety, data ethics, and responsibility, in this paper they will be grouped according to eight philosophical themes: Responsible Innovation, road justice, road safety, freedom, human control, privacy, data fairness, responsibility. These are proposed as the first topics for a new ethics of transportation.

The notion of “responsibility gap” with artificial intelligence (AI) was originally introduced in the philosophical debate to indicate the concern that “learning automata” may make more difficult or impossible to attribute moral culpability to persons for untoward events. Building on literature in moral and legal philosophy, and ethics of technology, the paper proposes a broader and more comprehensive analysis of the responsibility gap. The responsibility gap, it is argued, is not one problem but a set of at least four interconnected problems – gaps in culpability, moral and public accountability, active responsibility—caused by different sources, some technical, other organisational, legal, ethical, and societal. Responsibility gaps may also happen with non-learning systems. The paper clarifies which aspect of AI may cause which gap in which form of responsibility, and why each of these gaps matter. It proposes a critical review of partial and non-satisfactory attempts to address the responsibility gap: those which present it as a new and intractable problem (“fatalism”), those which dismiss it as a false problem (“deflationism”), and those which reduce it to only one of its dimensions or sources and/or present it as a problem that can be solved by simply introducing new technical and/or legal tools (“solutionism”). The paper also outlines a more comprehensive approach to address the responsibility gaps with AI in their entirety, based on the idea of designing socio-technical systems for “meaningful human control", that is systems aligned with the relevant human reasons and capacities.

In this article the title and running title were incorrectly given as ‘Marc Coeckelbergh, AI Ethics, Mit Press, 2021’ but should have been ’Mark Coeckelbergh, AI Ethics, Mit Press, 2021’. The original article has been corrected.

Accountability and responsibility are key concepts in the academic and societal debate on Autonomous Weapon Systems, but these notions are often used as high-level overarching constructs and are not operationalised to be useful in practice. “Meaningful Human Control” is often mentioned as a requirement for the deployment of Autonomous Weapon Systems, but a common definition of what this notion means in practice, and a clear understanding of its relation with responsibility and accountability is also lacking. In this paper, we present a definition of these concepts and describe the relations between accountability, responsibility, control and oversight in order to show how these notions are distinct but also connected. We focus on accountability as a particular form of responsibility—the obligation to explain one’s action to a forum—and we present three ways in which the introduction of Autonomous Weapon Systems may create “accountability gaps”. We propose a Framework for Comprehensive Human Oversight based on an engineering, socio-technical and governance perspective on control. Our main claim is that combining the control mechanisms at technical, socio-technical and governance levels will lead to comprehensive human oversight over Autonomous Weapon Systems which may ensure solid controllability and accountability for the behaviour of Autonomous Weapon Systems. Finally, we give an overview of the military control instruments that are currently used in the Netherlands and show the applicability of the comprehensive human oversight Framework to Autonomous Weapon Systems. Our analysis reveals two main gaps in the current control mechanisms as applied to Autonomous Weapon Systems. We have identified three first options as future work for the design of a control mechanism, one in the technological layer, one in the socio-technical layer and one the governance layer, in order to achieve comprehensive human oversight and ensure accountability over Autonomous Weapon Systems.

Increased on-road testing and market availability of partially automated vehicles (AV) offers researchers and developers the opportunity to evaluate the AV’s performance. The occurrence of new types of accidents involving AV’s has sparked questions in regard to who is actually in control over and responsible for AV control. In this contribution, we suggest a potential discrepancy in AV control with the review of recently documented accidents involving AV’s. The identification of a gap in control is performed using a recently formulated moral philosophical framework of Meaningful Human Control (MHC). This shows a discrepancy between the attribution of responsibility and the ability of a human to fulfil the role assigned to them. While a gap in control is not evident from the viewpoint of operational control, it requires the more intricate concept of MHC to expose it. Recommendations are further made that AV developers and vehicle approval authorities should consider control from a MHC perspective to avoid future gaps in control with the resulting consequences.

In this paper, in line with the general framework of value-sensitive design, we aim to operationalize the general concept of “Meaningful Human Control” (MHC) in order to pave the way for its translation into more specific design requirements. In particular, we focus on the operationalization of the first of the two conditions (Santoni de Sio and Van den Hoven 2018) investigated: the so-called ‘tracking’ condition. Our investigation is led in relation to one specific subcase of automated system: dual-mode driving systems (e.g. Tesla ‘autopilot’). First, we connect and compare meaningful human control with a concept of control very popular in engineering and traffic psychology (Michon 1985), and we explain to what extent tracking resembles and differs from it. This will help clarifying the extent to which the idea of meaningful human control is connected to, but also goes beyond, current notions of control in engineering and psychology. Second, we take the systematic analysis of practical reasoning as traditionally presented in the philosophy of human action (Anscombe, Bratman, Mele) and we adapt it to offer a general framework where different types of reasons and agents are identified according to their relation to an automated system’s behaviour. This framework is meant to help explaining what reasons and what agents (should) play a role in controlling a given system, thereby enabling policy makers to produce usable guidelines and engineers to design systems that properly respond to selected human reasons. In the final part, we discuss a practical example of how our framework could be employed in designing automated driving systems.

Automated driving systems (ADS) with partial automation are currently available for the consumer. They are potentially beneficial to traffic flow, fuel consumption, and safety, but human behaviour whilst driving with ADS is poorly understood. Human behaviour is currently expected to lead to dangerous circumstances as ADS could place human drivers ‘out-of-the-loop’ or cause other types of adverse behavioural adaptation. This article introduces the concept of ‘meaningful human control’ to better address the challenges raised by ADS, and presents a new framework of human control over ADS by means of literature-based categorisation. Using standards set by European authorities for driver skills and road rules, this framework offers a unique, quantified perspective into the effects of ADS on human behaviour. One main result is a rapid and inconsistent decrease in required skill- and rule-based behaviour mismatching with the increasing amount of required knowledge-based behaviour. Furthermore, the development of higher levels of automation currently requires different human behaviour than feasible, as a mismatch between supply and demand in terms of behaviour arises. Implications, discrepancies and emerging mismatches this framework elicits are discussed, and recommendations towards future design strategies and research opportunities are made to provide a meaningful transition of human control over ADS.

This paper proposes that autonomous vehicles should be designed to reduce light pollution. In support of this specific proposal, a moral assessment of autonomous vehicles more comprehensive than the dilemmatic life-and-death questions of trolley problem-style situations is presented. The paper therefore consists of two interre-lated arguments. The first is that autonomous vehicles are currently still a technol-ogy in development, and not one that has acquired its definitive shape, meaning the design of both the vehicles and the surrounding infrastructure is open-ended. Design for values is utilized to articulate a path forward, by which engineering ethics should strive to incorporate values into a technology during its development phase. Second, it is argued that nighttime lighting—a critical supporting infrastructure—should be a prima facie consideration for autonomous vehicles during their development phase. It is shown that a reduction in light pollution, and more boldly a better balance of lighting and darkness, can be achieved via the design of future autonomous vehicles. Two case studies are examined (parking lots and highways) through which autono-mous vehicles may be designed for “driving in the dark.” Nighttime lighting issues are thus inserted into a broader ethics of autonomous vehicles, while simultaneously introducing questions of autonomous vehicles into debates about light pollution.

In the political debate on Autonomous Weapon Systems strong views and opinions are voiced, but empirical research to support these opinions is lacking. Insight into which moral values are related to the deployment of Autonomous Weapon Systems is missing. We describe the empirical results of two studies on moral values regarding Autonomous Weapon Systems that aim to understand the perception of people pertaining to the introduction of Autonomous Weapon Systems. One study consists of a sample of military personnel of the Dutch Ministry of Defense and the second study contains a sample of civilians. The results indicate both groups are more anxious about the deployment of Autonomous Weapon Systems than about the deployment of Human Operated drones, and that they perceive Autonomous Weapon Systems to have less respect for the dignity of human life. The concerns for Autonomous Weapon Systems creating new kinds of psychological and moral harm is very present in the public debate, and this is in our opinion one element that deserves to be carefully considered in future debates on the ethics of the design and deployment of Autonomous Weapon Systems. The results of these studies reveal a common ground regarding the moral values of human dignity and anxiety pertaining the introduction of Autonomous Weapon Systems which could further the ethical debate.

Personalized medicine uses fine grained information on individual persons, to pinpoint deviations from the normal. ‘Digital Twins’ in engineering provide a conceptual framework to analyze these emerging data-driven health care practices, as well as their conceptual and ethical implications for therapy, preventative care and human enhancement. Digital Twins stand for a specific engineering paradigm, where individual physical artifacts are paired with digital models that dynamically reflects the status of those artifacts. When applied to persons, Digital Twins are an emerging technology that builds on in silico representations of an individual that dynamically reflect molecular status, physiological status and life style over time. We use Digital Twins as the hypothesis that one would be in the possession of very detailed bio-physical and lifestyle information of a person over time. This perspective redefines the concept of ‘normality’ or ‘health,’ as a set of patterns that are regular for a particular individual, against the backdrop of patterns observed in the population. This perspective also will impact what is considered therapy and what is enhancement, as can be illustrated with the cases of the ‘asymptomatic ill’ and life extension via anti-aging medicine. These changes are the consequence of how meaning is derived, in case measurement data is available. Moral distinctions namely may be based on patterns found in these data and the meanings that are grafted on these patterns. Ethical and societal implications of Digital Twins are explored. Digital Twins imply a data-driven approach to health care. This approach has the potential to deliver significant societal benefits, and can function as a social equalizer, by allowing for effective equalizing enhancement interventions. It can as well though be a driver for inequality, given the fact that a Digital Twin might not be an accessible technology for everyone, and given the fact that patterns identified across a population of Digital Twins can lead to segmentation and discrimination. This duality calls for governance as this emerging technology matures, including measures that ensure transparency of data usage and derived benefits, and data privacy.

Debates on lethal autonomous weapon systems have proliferated in the past 5 years. Ethical concerns have been voiced about a possible raise in the number of wrongs and crimes in military operations and about the creation of a “responsibility gap” for harms caused by these systems. To address these concerns, the principle of “meaningful human control” has been introduced in the legal–political debate; according to this principle, humans not computers and their algorithms should ultimately remain in control of, and thus morally responsible for, relevant decisions about (lethal) military operations. However, policy-makers and technical designers lack a detailed theory of what “meaningful human control” exactly means. In this paper, we lay the foundation of a philosophical account of meaningful human control, based on the concept of “guidance control” as elaborated in the philosophical debate on free will and moral responsibility. Following the ideals of “Responsible Innovation” and “Value-sensitive Design,” our account of meaningful human control is cast in the form of design requirements. We identify two general necessary conditions to be satisfied for an autonomous system to remain under meaningful human control: first, a “tracking” condition, according to which the system should be able to respond to both the relevant moral reasons of the humans designing and deploying the system and the relevant facts in the environment in which the system operates; second, a “tracing” condition, according to which the system should be designed in such a way as to grant the possibility to always trace back the outcome of its operations to at least one human along the chain of design and operation. As we think that meaningful human control can be one of the central notions in ethics of robotics and AI, in the last part of the paper, we start exploring the implications of our account for the design and use of non-military autonomous systems, for instance, self-driving cars.