This handbook presents the concept of ‘meaningful human control’ (MHC) over AI systems from the perspectives of (i) philosophy and ethics, (ii) law and governance, and (iii) design and engineering. The introductory chapter addresses the motivations and recent developments in MHC, introducing each perspective and related chapters. These three disciplinary perspectives scrutinize how MHC intertwines with philosophical debates on moral responsibility, societal concerns regarding control over technological advancements in legal frameworks, and the engineering complexities of designing and developing AI systems while ensuring human control and responsibility. Additionally, cross-cutting aspects on MHC over AI systems are also introduced and discussed through (iv) interdisciplinary and systemic perspectives. By offering a contextualized introduction to the perspectives considered in this handbook, this chapter aims to present the handbook’s various approaches and points of interest for a diverse audience, highlighting potential entry points into this multidisciplinary volume.

The urban traffic environment is characterized by the presence of a highly differentiated pool of users, including vulnerable ones. This makes vehicle automation particularly difficult to implement, as a safe coordination among those users is hard to achieve in such an open scenario. Different strategies have been proposed to address these coordination issues, but all of them have been found to be costly for they negatively affect a range of human values (e.g. safety, democracy, accountability…). In this paper, we claim that the negative value impacts entailed by each of these strategies can be interpreted as lack of what we call Meaningful Human Control over different parts of a sociotechnical system. We argue that Meaningful Human Control theory provides the conceptual tools to reduce those unwanted consequences, and show how “designing for meaningful human control” constitutes a valid strategy to address coordination issues. Furthermore, we showcase a possible application of this framework in a highly dynamic urban scenario, aiming to safeguard important values such as safety, democracy, individual autonomy, and accountability. Our meaningful human control framework offers a perspective on coordination issues that allows to keep human actors in control while minimizing the active, operational role of the drivers. This approach makes ultimately possible to promote a safe and responsible transition to full automation.

Automated vehicles (AVs) aim to dramatically improve traffic safety by reducing or eliminating human error, which remains the leading cause of road crashes. However, commonly accepted standards for the ‘safe driving behaviour of machines’ are pending and urgently needed. Unless a common understanding of safety as a design value is achieved, different manufacturers’ driving styles may emerge, resulting in inconsistent, unpredictable and potentially unsafe ‘behaviour’ of AVs in certain situations. This paper aims to explore the main gaps and challenges towards establishing shared safety standards for the ‘behaviour’ of AVs, and contribute to their responsible traffic integration, by reviewing the state-of-the-art on AV safety in the core relevant disciplines: ethics of technology, safety science (engineering & human factors), and standardisation. The ethical and safety aspects investigated include the users’ perception of AV safety, the ethical trade-offs in critical decision-making contexts, the pertinence of data-driven approaches for AVs to mimic human behaviour, and the responsibilities of various actors. Moreover, the paper reviews the current safety patterns, metrics (surrogate measures of safety – SMoS) and their thresholds introduced in existing research for three use cases: mixed traffic of AV and conventional vehicles, AV interaction with pedestrians and cyclists, and transition of control from machine to human driver. The results reveal several knowledge gaps within each discipline and highlights the lack of common understanding of safety across disciplines. On the basis of the results, the paper proposes a framework for further research on AV safety, identifying concrete opportunities for interdisciplinary research, with common goals and methodologies, and explicitly indicating the path for transfer of knowledge between sectors.

The paper has two goals. The first is presenting the main results of the recent report Ethics of Connected and Automated Vehicles: recommendations on road safety, privacy, fairness, explainability and responsibility written by the Horizon 2020 European Commission Expert Group to advise on specific ethical issues raised by driverless mobility, of which the author of this paper has been member and rapporteur. The second is presenting some broader ethical and philosophical implications of these recommendations, and using these to contribute to the establishment of Ethics of Transportation as an independent branch of applied ethics. The recent debate on the ethics of Connected and Automated Vehicles (CAVs) presents a paradox and an opportunity. The paradox is the presence of a flourishing debate on the ethics of one very specific transportation technology without ethics of transportation being in itself a well-established academic discipline. The opportunity is that now that a spotlight has been switched on the ethical dimensions of CAVs it may be easier to establish a broader debate on ethics of transportation. While the 20 recommendations of the EU report are grouped in three macro-areas: road safety, data ethics, and responsibility, in this paper they will be grouped according to eight philosophical themes: Responsible Innovation, road justice, road safety, freedom, human control, privacy, data fairness, responsibility. These are proposed as the first topics for a new ethics of transportation.

Accountability and responsibility are key concepts in the academic and societal debate on Autonomous Weapon Systems, but these notions are often used as high-level overarching constructs and are not operationalised to be useful in practice. “Meaningful Human Control” is often mentioned as a requirement for the deployment of Autonomous Weapon Systems, but a common definition of what this notion means in practice, and a clear understanding of its relation with responsibility and accountability is also lacking. In this paper, we present a definition of these concepts and describe the relations between accountability, responsibility, control and oversight in order to show how these notions are distinct but also connected. We focus on accountability as a particular form of responsibility—the obligation to explain one’s action to a forum—and we present three ways in which the introduction of Autonomous Weapon Systems may create “accountability gaps”. We propose a Framework for Comprehensive Human Oversight based on an engineering, socio-technical and governance perspective on control. Our main claim is that combining the control mechanisms at technical, socio-technical and governance levels will lead to comprehensive human oversight over Autonomous Weapon Systems which may ensure solid controllability and accountability for the behaviour of Autonomous Weapon Systems. Finally, we give an overview of the military control instruments that are currently used in the Netherlands and show the applicability of the comprehensive human oversight Framework to Autonomous Weapon Systems. Our analysis reveals two main gaps in the current control mechanisms as applied to Autonomous Weapon Systems. We have identified three first options as future work for the design of a control mechanism, one in the technological layer, one in the socio-technical layer and one the governance layer, in order to achieve comprehensive human oversight and ensure accountability over Autonomous Weapon Systems.

In the political debate on Autonomous Weapon Systems strong views and opinions are voiced, but empirical research to support these opinions is lacking. Insight into which moral values are related to the deployment of Autonomous Weapon Systems is missing. We describe the empirical results of two studies on moral values regarding Autonomous Weapon Systems that aim to understand the perception of people pertaining to the introduction of Autonomous Weapon Systems. One study consists of a sample of military personnel of the Dutch Ministry of Defense and the second study contains a sample of civilians. The results indicate both groups are more anxious about the deployment of Autonomous Weapon Systems than about the deployment of Human Operated drones, and that they perceive Autonomous Weapon Systems to have less respect for the dignity of human life. The concerns for Autonomous Weapon Systems creating new kinds of psychological and moral harm is very present in the public debate, and this is in our opinion one element that deserves to be carefully considered in future debates on the ethics of the design and deployment of Autonomous Weapon Systems. The results of these studies reveal a common ground regarding the moral values of human dignity and anxiety pertaining the introduction of Autonomous Weapon Systems which could further the ethical debate.

Automated driving systems (ADS) with partial automation are currently available for the consumer. They are potentially beneficial to traffic flow, fuel consumption, and safety, but human behaviour whilst driving with ADS is poorly understood. Human behaviour is currently expected to lead to dangerous circumstances as ADS could place human drivers ‘out-of-the-loop’ or cause other types of adverse behavioural adaptation. This article introduces the concept of ‘meaningful human control’ to better address the challenges raised by ADS, and presents a new framework of human control over ADS by means of literature-based categorisation. Using standards set by European authorities for driver skills and road rules, this framework offers a unique, quantified perspective into the effects of ADS on human behaviour. One main result is a rapid and inconsistent decrease in required skill- and rule-based behaviour mismatching with the increasing amount of required knowledge-based behaviour. Furthermore, the development of higher levels of automation currently requires different human behaviour than feasible, as a mismatch between supply and demand in terms of behaviour arises. Implications, discrepancies and emerging mismatches this framework elicits are discussed, and recommendations towards future design strategies and research opportunities are made to provide a meaningful transition of human control over ADS.

In this paper, in line with the general framework of value-sensitive design, we aim to operationalize the general concept of “Meaningful Human Control” (MHC) in order to pave the way for its translation into more specific design requirements. In particular, we focus on the operationalization of the first of the two conditions (Santoni de Sio and Van den Hoven 2018) investigated: the so-called ‘tracking’ condition. Our investigation is led in relation to one specific subcase of automated system: dual-mode driving systems (e.g. Tesla ‘autopilot’). First, we connect and compare meaningful human control with a concept of control very popular in engineering and traffic psychology (Michon 1985), and we explain to what extent tracking resembles and differs from it. This will help clarifying the extent to which the idea of meaningful human control is connected to, but also goes beyond, current notions of control in engineering and psychology. Second, we take the systematic analysis of practical reasoning as traditionally presented in the philosophy of human action (Anscombe, Bratman, Mele) and we adapt it to offer a general framework where different types of reasons and agents are identified according to their relation to an automated system’s behaviour. This framework is meant to help explaining what reasons and what agents (should) play a role in controlling a given system, thereby enabling policy makers to produce usable guidelines and engineers to design systems that properly respond to selected human reasons. In the final part, we discuss a practical example of how our framework could be employed in designing automated driving systems.