E.P. Verdiesen | TU Delft Repository

Comprehensive Human Oversight over Autonomous Weapon Systems

Doctoral thesis (2024) - E.P. Verdiesen

Value elicitation on a scenario of autonomous weapon system deployment

a qualitative study based on the value deliberation process

Journal article (2023) - E.P. Verdiesen, M.V. Dignum

Ethical concerns on autonomous weapon systems (AWS) call for a process of human oversight to ensure accountability over targeting decisions and the use of force. To align the behavior of autonomous systems with human values and norms, the Design for Values approach can be used to consciously embody values in the deployment of AWS. One instrument for the elicitation of values during the design is participative deliberation. In this paper, we describe a participative deliberation method and results of a value elicitation by means of the value deliberation process for which we organized two panels each consisting of a mixture of experts in the field of AWS working in military operations, foreign policy, NGO’s and industry. The results of our qualitative study indicate not only that value discussion leads to changes in perception of the acceptability of alternatives, or options, in a scenario of AWS deployment, it also gives insight in to which values are deemed important and highlights that trust in the decision-making of an AWS is crucial. ...

Comprehensive Human Oversight Framework to Ensure Accountability over Autonomous Weapon Systems

Conference paper (2022) - E.P. Verdiesen

Integrating Comprehensive Human Oversight in Drone Deployment

A Conceptual Framework Applied to the Case of Military Surveillance Drones

Journal article (2021) - E.P. Verdiesen, Andrea Aler Tubella, M.V. Dignum

Accountability is a value often mentioned in the debate on intelligent systems and their increased pervasiveness in our society. When focusing specifically on autonomous systems, a critical gap emerges: although there is much work on governance and attribution of accountability, there is a significant lack of methods for the operationalisation of accountability within the socio-technical layer of autonomous systems. In the case of autonomous unmanned aerial vehicles- or drones—the critical question of how to maintain accountability as they undertake fully autonomous flights becomes increasingly important as their uses multiply in both the commercial and military fields. In this paper, we aim to fill the operationalisation gap by proposing a socio-technical framework to guarantee human oversight and accountability in drone deployments, showing its enforceability in the real case of military surveillance drones. By keeping a focus on accountability and human oversight as values, we align with the emphasis placed on human responsibility, while requiring a concretisation of what these principles mean for each specific application, connecting them with concrete socio-technical requirements. In addition, by constraining the framework to observable elements of pre- and post-deployment, we do not rely on assumptions made on the internal workings of the drone nor the technical fluency of the operator. ...

Accountability and Control Over Autonomous Weapon Systems

A Framework for Comprehensive Human Oversight

Journal article (2020) - Ilse Verdiesen, Filippo Santoni de Sio, Virginia Dignum

Accountability and responsibility are key concepts in the academic and societal debate on Autonomous Weapon Systems, but these notions are often used as high-level overarching constructs and are not operationalised to be useful in practice. “Meaningful Human Control” is often mentioned as a requirement for the deployment of Autonomous Weapon Systems, but a common definition of what this notion means in practice, and a clear understanding of its relation with responsibility and accountability is also lacking. In this paper, we present a definition of these concepts and describe the relations between accountability, responsibility, control and oversight in order to show how these notions are distinct but also connected. We focus on accountability as a particular form of responsibility—the obligation to explain one’s action to a forum—and we present three ways in which the introduction of Autonomous Weapon Systems may create “accountability gaps”. We propose a Framework for Comprehensive Human Oversight based on an engineering, socio-technical and governance perspective on control. Our main claim is that combining the control mechanisms at technical, socio-technical and governance levels will lead to comprehensive human oversight over Autonomous Weapon Systems which may ensure solid controllability and accountability for the behaviour of Autonomous Weapon Systems. Finally, we give an overview of the military control instruments that are currently used in the Netherlands and show the applicability of the comprehensive human oversight Framework to Autonomous Weapon Systems. Our analysis reveals two main gaps in the current control mechanisms as applied to Autonomous Weapon Systems. We have identified three first options as future work for the design of a control mechanism, one in the technological layer, one in the socio-technical layer and one the governance layer, in order to achieve comprehensive human oversight and ensure accountability over Autonomous Weapon Systems. ...

Accountability and responsibility are key concepts in the academic and societal debate on Autonomous Weapon Systems, but these notions are often used as high-level overarching constructs and are not operationalised to be useful in practice. “Meaningful Human Control” is often mentioned as a requirement for the deployment of Autonomous Weapon Systems, but a common definition of what this notion means in practice, and a clear understanding of its relation with responsibility and accountability is also lacking. In this paper, we present a definition of these concepts and describe the relations between accountability, responsibility, control and oversight in order to show how these notions are distinct but also connected. We focus on accountability as a particular form of responsibility—the obligation to explain one’s action to a forum—and we present three ways in which the introduction of Autonomous Weapon Systems may create “accountability gaps”. We propose a Framework for Comprehensive Human Oversight based on an engineering, socio-technical and governance perspective on control. Our main claim is that combining the control mechanisms at technical, socio-technical and governance levels will lead to comprehensive human oversight over Autonomous Weapon Systems which may ensure solid controllability and accountability for the behaviour of Autonomous Weapon Systems. Finally, we give an overview of the military control instruments that are currently used in the Netherlands and show the applicability of the comprehensive human oversight Framework to Autonomous Weapon Systems. Our analysis reveals two main gaps in the current control mechanisms as applied to Autonomous Weapon Systems. We have identified three first options as future work for the design of a control mechanism, one in the technological layer, one in the socio-technical layer and one the governance layer, in order to achieve comprehensive human oversight and ensure accountability over Autonomous Weapon Systems.

Moral Values Related to Autonomous Weapon Systems

An Empirical Survey that Reveals Common Ground for the Ethical Debate

Journal article (2019) - Ilse Verdiesen, Filippo Santoni De Sio, Virginia Dignum

In the political debate on Autonomous Weapon Systems strong views and opinions are voiced, but empirical research to support these opinions is lacking. Insight into which moral values are related to the deployment of Autonomous Weapon Systems is missing. We describe the empirical results of two studies on moral values regarding Autonomous Weapon Systems that aim to understand the perception of people pertaining to the introduction of Autonomous Weapon Systems. One study consists of a sample of military personnel of the Dutch Ministry of Defense and the second study contains a sample of civilians. The results indicate both groups are more anxious about the deployment of Autonomous Weapon Systems than about the deployment of Human Operated drones, and that they perceive Autonomous Weapon Systems to have less respect for the dignity of human life. The concerns for Autonomous Weapon Systems creating new kinds of psychological and moral harm is very present in the public debate, and this is in our opinion one element that deserves to be carefully considered in future debates on the ethics of the design and deployment of Autonomous Weapon Systems. The results of these studies reveal a common ground regarding the moral values of human dignity and anxiety pertaining the introduction of Autonomous Weapon Systems which could further the ethical debate. ...

The design of human oversight in autonomous weapon systems

Conference paper (2019) - Ilse Verdiesen

Autonomous Weapon Systems (AWS) can be defined as weapons systems equipped with Artificial Intelligence (AI). They are an emerging technology and are increasingly deployed on the battlefield. In the societal debate on Autonomous Weapon Systems, the concept of Meaningful Human Control (MHC) is often mentioned as requirement, but MHC will not suffice as requirement to minimize unintended consequences of Autonomous Weapon Systems, because the definition of ‘control’ implies that one has the power to influence or direct the course of events or the ability to manage a machine. The characteristics autonomy, interactivity and adaptability of AI in Autonomous Weapon Systems inherently imply that control in strict sense is not possible. Therefore, a different approach is needed to minimize unintended consequences of AWS. Several scholars are describing the concept of Human Oversight in Autonomous Weapon Systems and AI in general. Just recently Taddeo and Floridi (2018) describe that human oversight procedures are necessary to minimize unintended consequences and to compensate unfair impacts of AI. In my PhD project, I will analyse the concepts that are needed to define, model, evaluate and ensure human oversight in Autonomous Weapons and design a technical architecture to implement this. ...

The Design of Human Oversight in Autonomous Weapon Systems

Conference paper (2018) - Ilse Verdiesen

As the reach and capabilities of Artificial Intelligence (AI) systems increases, there is also a growing awareness of the ethical, legal and societal impact of the potential actions and decisions of these systems. Many are calling for guidelines and regulations that can ensure the responsible design, development, implementation, and policy of AI. In scientific literature, AI is characterized by the concepts of Adaptability, Interactivity and Autonomy (Floridi & Sanders, 2004). According to Floridi and Sanders (2004), Adaptability means that the system can change based on its interaction and can learn from its experience. Machine learning techniques are an example of this. Interactivity occurs when the system and its environment act upon each other and Autonomy implies that the system itself can change its state. ...

MOOD: Massive Open Online Deliberation Platform-A Practical Application

Conference paper (2016) - E.P. Verdiesen, Virginia Dignum, Jeroen van den Hoven, Martijn Cligge, Jan Timmermans, Lennard Segers

Nowadays, public debates often take place on social media platforms like Facebook or Twitter and can be characterized as asynchronous, protracted and ill-structured. The Massive Open Online Deliberation (MOOD) platform aims to structure these debates. Essential is that the platform can differentiate between the moral acceptability and the social acceptance of a debate outcome. We briefly describe the e-deliberation process and look at two existing debate platforms, Liquidfeedback and Debatehub. We design and build a prototype that mainly focuses on:(1) a method to differentiate and validate facts and opinions, and (2) a mechanism that maps both the social acceptance and the moral acceptability of debate outcomes. We research these ethical concepts more in depth and implement several techniques, such as a voting mechanism, in a working prototype that supports a four stage deliberation process. In future applications, machine learning techniques can be integrated in the platform to perform sentiment analysis on a debate. ...