Scandals in which governmental ADM tools played a role, have recently brought about political and societal debate about the potential harms to citizens that such automated systems potentially bring. This thesis focuses on ADM systems which contain an AI component. Based on the knowledge gaps perceived, the main research question for this study is: In public sector AI systems, what are emerging vulnerabilities for citizens and how do these translate into governance requirements for decision-makers? The approach to answer this question is an adjusted form of Theory Building from Case Study. A layered ’onion model’ presents four relevant contexts to consider for AI System vulnerabilities specifically in the public sector: AI model, Model deployment, Political-administrative, and Societal. The case study results demonstrate that dealing with vulnerabilities in one of the four model contexts often complicates dealing with vulnerabilities in the other contexts. Hence, the vulnerabilities model points to so-called governance requirements dilemmas.

There is a need for a more comprehensive sociotechnical systems view on ML. Such a view looks at the development and use of an ML system in practice as being a sociotechnical ML system: "a system consisting of technical artefacts, human agents and institutions, in which a machine-based subsystem influences its real or virtual environment by automating, supporting or augmenting decision-making". This research takes on this view to design a sociotechnical guide for ML practice, centring the specication of sociotechnical ML systems. Taking on the guidelines contributes to a safe and effective development and use of ML systems.

The use of Automated Decision Making (ADM) systems in the public sector will become increasingly prevalent in the future, making citizens increasingly likely to be confronted with decisions that have been made fully automatically, without human intervention. Ever more digitization increases the opaqueness of the social benefits system in the Netherlands and leads to citizens facing most of the adverse, sometimes unforeseen effects of automation. Why and how unsafeness and unexpected effects emerge is not well known. In this research, these effects, defined to be ripple effects, are explored using a system safety lens and proposing an agent-based model design to explore why and how the effects emerge. The leading cause of ripple effects was found to be the lack of a hierarchical safety structure, seen in laws being organized per organization, and the lack of an overarching controller to ensure that overlap does not occur. This is partly due to a lack of an accurate model representing a Dutch citizen. Several policy recommendations are presented, including a rudimentary set-up on how to apply a safety control structure within the social benefits system.

In our thesis, we conducted an in-depth exploration of the integration of Artificial Intelligence (AI) in public sector decision-making, focusing particularly on AI case routing for welfare benefits allocation. Initially, our research was guided by a techno-optimistic viewpoint, presuming that municipalities would inherently benefit from AI adoption. However, as our study progressed, we encountered the complexities and challenges inherent in the practical implementation of AI in public services. We employed the Design Science Research methodology, utilizing a diverse array of methods including literature reviews, case studies, expert interviews, and system safety analysis. Our objective was to develop a method-type artifact to enhance decision-making in civil services, with a special emphasis on the application of AI in welfare benefits allocation. Our findings indicated that AI implementation in the public sector is highly context-dependent, requiring a tailored approach that addresses various technical, organizational, and cultural barriers. Throughout our research, we identified critical factors such as the need for transparency, fairness, and accountability in AI systems. We recognized the importance of adopting a balanced perspective that considers both AI and non-AI solutions. Our study highlighted the potential benefits of AI in public services, such as increased efficiency and improved quality of decision-making. However, we also noted significant challenges in ensuring system safety and maintaining ethical standards. Through expert interviews and a detailed system safety analysis, we emphasized the necessity of establishing clear rules for the responsible use of AI case routing and addressing the associated risks. Our research concluded that AI might not always be the most suitable approach, particularly if safety concerns and ethical considerations outweigh the potential benefits. In summary, our thesis underscores the need for a critical and balanced approach to the integration of AI in public services. We advocate for a holistic strategy that involves collaboration among government, academia, industry, and civil society. This approach is vital to harness the potential benefits of AI while mitigating risks and ensuring the well-being of society.

Currently, the electricity grid in the Netherlands is reaching its capacity, resulting in congestion issues. One of the factors that causes the electricity grid to become overloaded is the increasing use of EVs (electric vehicles). A situation in which a large number of EV users in a certain area charge their EV at the same time can significantly increase the risk of congestion in the electricity network. To avoid such situations it is necessary to change the charging behaviour of the EV users. The literature shows that smart charging systems, such as charging schemes, are having a high potential to solve these grid capacity problems. In the context of this study, a charging scheme is defined as a contract between an EV user and the charge card provider/electricity supplier stating, among others, at what times an EV user could charge his/her EV. However, to ensure that charging schemes can effectively contribute to solving the grid capacity problems, it is important that many EV users are willing to participate in such schemes. Therefore, the research objective is to study how EV users will react to various charging schemes. In particular, the aim is to assess the effectiveness of appealing to someone's intrinsic motivation versus providing extrinsic incentives. First of all, with regard to the extrinsic incentives, this study specifically concerns monetary incentives. By presenting different discounts, it is possible to test whether a monetary compensation influences the willingness to participate in a charging scheme. Secondly, with regard to someone’s intrinsic motivation, this study focuses on appealing to someone's solidarity as a form of an intrinsic motivation. EV users with a certain degree of solidarity can be motivated to participate in a charging scheme when solidarity incentives are obtained (social recognition, togetherness and the development of friendships). By emphasizing the social benefits of a charging scheme, solidarity incentives could possibly increase the willingness to participate in a charging scheme. Since the objective of the study is to research how EV users will react to various charging schemes, a survey containing a discrete choice experiment is used. To be able to examine the effectiveness of providing an extrinsic incentive (monetary compensation) versus appealing to someone's solidarity by emphasizing the social benefits of a charging scheme, the survey contains two charging scheme versions, an incentive-based version and an intrinsic-based version. By using two survey versions, it is possible to examine separately whether it is more effective to appeal to someone's intrinsic motivation versus providing a financial compensation. The results show that the willingness to participate in a charging scheme was generally higher in the intrinsic-based version than in the incentive-based version. Based on this, it can be concluded, within the boundaries of the selected attribute ranges, that appealing to someone's solidarity by emphasizing the social benefits is more effective than offering a financial compensation. This is also substantiated by the fact that the financial compensation attribute included in the incentive-based charging schemes does not influence the willingness to participate.

The electricity grid in the Netherlands is highly congested and poses a significant challenge due to increasing demand and the transition to sustainable energy sources, which overwhelms the existing infrastructure. To address this issue, Distribution System Operators (DSOs) like Alliander are tasked with managing waiting lists for power connections and transport capacity requests. The traditional First-Come-First-Serve (FCFS) principle unintentionally delays transport requests that can be argued are more important to society. This motivated the policymakers to propose a prioritization framework based on societal importance. This framework prioritizes clients in categories such as congestion softeners, safety, basic needs, and sustainability, aiming to benefit society as a whole. However, the introduction of this framework raises several implications, particularly concerning distributive justice. To systematically identify and deliberate these implications, a Design Science Research methodology was used. This methodology involved a comprehensive literature review across various research areas including Energy Justice, Waiting List Management, and Distributive Justice. Subsequently, a qualitative and quantitative analysis was conducted to understand the impacts on waiting times for both priority and non-priority clients. Stakeholder opinions were analyzed across different industries, revealing concerns and suggestions regarding the social prioritization framework. The quantitative analysis estimated the real impact on waiting lists, considering technical constraints and proposing distributive justice metrics such as time equity disparity and change in the estimated time-to-solution ratio. The analysis of real waiting lists demonstrated mixed outcomes, with priority clients experiencing decreases in waiting times while non-priority clients faced delays. Some non-priority clients were needlessly delayed. Distributive justice metrics highlighted disproportionate burdens on non-priority clients. Moreover, the framework's impacts extended beyond delays to financial, and psychological effects. Recommendations for future research include incorporating historical data, obtaining more client information, improving stakeholder feedback loops, and further analyzing other aspects of Energy Justice. Additionally, connecting quantitative implications to societal values could enhance understanding of the framework's benefits to society. Overall, this research provides valuable insights into the implications of implementing a social prioritization framework for electricity grid waiting lists, highlighting the importance of considering distributive justice and societal impacts in policy development and decision-making processes.

Explainable AI is the field concerned with trying to make AI understandable to humans. While efforts have resulted in significant improvement in research and practical methods of Explainable AI, there is an urgent need for additional research and empirical studies. The academic research gaps identified in this thesis show that Explainable AI is still in its infancy and is mostly approached with a technocentric perspective while not being focused on the audience the explainability is actually intended for. Next to this, there is no structured approach to defining and establishing explainability in dynamic complex systems that involve people, institutional, and organizational elements. Lastly, there are limited empirical studies that investigate the needs, usage, and risk of explainability in complex systems. The research tries to define and address explainability in the socio-technical context within the Machine Learning decision support systems of Transaction Monitoring. It does so by performing an extensive literature review and by conducting semi-structured that try to collect empirical knowledge within local practice. The goal of this research is to expand the definitions and view on explainability by incorporating the social, organizational, and institutional elements that influence explainability. Next, the goal is to develop a method that can help practitioners approach explainability in a structured manner taking the audience into account while applying this socio-technical perspective. The user-centered method for operationalizing explainability takes on a socio-technical perspective and can provide requirements for design choices, in addition to this the method shows how these requirements can be satisfied and controlled by instantiating control structures. The method has been demonstrated and evaluated within the bank by providing a workshop using a Toy Case during a focus group. Practitioners experienced the method as useful and actionable, also the method provides broader perspectives and insights on explainability and invites the discussion of dilemmas and questions. The practitioners added that the method could be further refined by focusing on additional guidance on the control structure because the method assumes prerequisite knowledge of systems theory and system safety theory.

Integrating artificial intelligence (AI) into Japan's life insurance sector marks a significant move towards data-centric precision, reflecting the nation's shift towards Society 5.0. In this domain, AI is revolutionizing decision-making processes and enhancing operational efficiency, with applications ranging from fraud detection in credit card systems to predictive underwriting. While AI offers notable benefits, it also introduces risks, including privacy breaches, algorithmic biases, and inadequate human supervision. To tackle these risks, the Japanese government has initiated guidelines for societal protection, but gaps remain in the insurance industry's implementation, especially in translating social norms into the industry's context to protect stakeholders' values. The industry needs a guide for safely designing, developing, and deploying AI systems, considering stakeholders' perspectives. This guide fills two knowledge gaps: a framework for translating high-level values into Japanese life insurance industry requirements and an initial process for converting these high-level values into organizational guidelines. An empirical study on predictive underwriting informed the research, identifying 13 values and four informal social institutions for the AI design process. It involved eight experts who defined 54 norms, which were later refined and categorized into process and assessment norms focusing on data and AI. The result is ten design guidelines for AI system developers, which are validated by experts, addressing the full AI lifecycle. These guidelines contribute scientifically by introducing an initial process combining design for values with system safety concepts, reporting standardization, and AI governance frameworks. Future research should replicate this process in various contexts, reevaluate the value framework with broader stakeholder inputs, investigate the dynamics between Japanese society and AI in more detail, and delve deeper into system theoretic hazards analysis. This approach promises to strengthen the value framework and process applicability in different organizational settings.

Governmental AI-supported decision-making is paramount when impacting citizens. Citizens are subject to their government’s decision-making, which is crucial when they transact, as examining the transaction’s rightfulness is executed by the same government. Thus, control and monitoring are evident, which are increasingly applied through AI-supported tools (Hoekstra et al., 2021). It is deemed socially unacceptable when government falsely accuses their citizens of unrightful transactions. Vice versa, it is also deemed unacceptable when criminals taking advantage of public money are exonerated. The citizens whom are trapped in these systems, like the benefits system, are often vulnerable and have low incomes. The Dutch government wrongly accused 20.000 parents of fraud, resulting in major con- sequences. Therefore, research is crucial for both a better functioning government and protecting the safety of vulnerable citizens. Scientific relevance is emphasised by the narrow understanding of how harm of next similar case is prevented, as such systems are barely researched holistically. Limited understanding of the relations within and between socio-technological contexts and the safekeeping of citizens lead to the main question: “How can citizen’s safety be safeguarded in governmental AI-supported decision-making?” Design science is the main methodology, consisting of three cycles: the rigour cycle, relevance cycle, and design cycle (Hevner, 2014). The method allows for innovative thinking and combining empirical (relevance) and scientific (rigour) knowledge toward exploring a new solution, in this research, safeguarding citizen’s safety (design). Additionally, it allows for a system’s approach required to understand the relationships of different decision-making components and for combining empirical and scientific insights (vom Brocke et al., 2020). Combining this with a holistic perspective, the research incorporates strategical, tactical, and operational challenges and specifically including the political dimension; more than factors and actors. Insight is created between the different context, which are all crucial in establishing a well balanced system. In seeking an answer to the main question, three parts are crucial: the system of AI-supported decision-making, the concept of citizen’s safety and how both can be integrated. The system and its boundaries are discovered through scientific and empirical exploration, serving as a framework. The definition of citizen’s safety and its implications are discovered by relating to the system and its characteristics. With these two parts, the influence of the system on citizen’s safety is explored and leads to the third crucial part, validated by a serious game. All parts use different sub-methodologies to obtain the required knowledge for the design science methodology.

To combat the effects of climate change, there is a worldwide shift to reduced emissions and increased use of renewable energy sources. Solar energy is a vital part of this transition and necessary to be able to achieve the targets. The rapid pace of adaptation raises questions about the negative equity and justice associated with this development. Along with the rapid growth, regional differences in solar panel adoption have been found across the world. Besides, government incentives and policies have played an important role in both adoption growth and justice implications. This study explores the pace and equitability of the solar energy transition in Amsterdam, the Netherlands. The study evaluates the potential for solar energy, the disparity of adoption, what socio-economic factors are correlated to adoption patterns, and how adoption might evolve in the future and under different policy measures. To do so, a structured, integrated and data-driven approach is designed to meet the research objectives, and possibly serve as a policy tool for future studies. The study shows that there is an "adoption gap" in Amsterdam, meaning that solar panels and the benefits of incentives end up with a specific group of citizens. Therefore, targeted policy measures are necessary to ensure an equitable transition.

This research looked into Algorithmic Systems used in the execution of the SUWI-law by executing agencies. It analysed the use and governance of these systems and their possible harms to citizens and society by means of an Actor Analysis and System-Theoretical Process Analysis (STPA), supported by semi-structured interviews. It aimed to answer the following research question: How can a systems safety perspective combined with an actor perspective be used to provide a sociotechnical description of algorithmic systems, their governance and possible hazards at agencies executing the SUWI-law? By means of the Actor Analysis, this research was able to describe the actor field and implications of it on the problem situation. The main insight from the actor field was the presence of ‘double binds’, several value conflicts that trouble use of AS. STPA, a method that focusses on safety of a system as a whole rather than individual components, provided the ability to map algorithmic systems within their sociotechnical context. It focussed on three hazards, which are system states that lead to accidents. These three hazards were the use of flawed logic in algorithmic systems, the use of flawed information in algorithmic system, and the flawed understanding citizens have of the role of the systems. They were explored through a Safety Control Structure, which displays controlled processes and those controlling them, and the construction of causal scenarios. Semi-structured interviews focussed on the governance of algorithmic systems, and possible improvements. They provided empirical input for STPA. The main contributions of this research are the proposal it makes for a sociotechnical description of algorithmic systems and their harms, using concepts derived from STPA. Such descriptions can be used to build shared understanding of algorithmic systems and their harms. It showed how such a description can be constructed using an Actor Analysis, interviews, and STPA, describing how these methods work together.

This research looked into Algorithmic Systems used in the execution of the SUWI-law by executing agencies. It analysed the use and governance of these systems and their possible harms to citizens and society by means of an Actor Analysis and System-Theoretical Process Analysis (STPA), supported by semi-structured interviews. It aimed to answer the following research question: How can a systems safety perspective combined with an actor perspective be used to provide a sociotechnical description of algorithmic systems, their governance and possible hazards at agencies executing the SUWI-law? By means of the Actor Analysis, this research was able to describe the actor field and implications of it on the problem situation. The main insight from the actor field was the presence of ‘double binds’, several value conflicts that trouble use of AS. STPA, a method that focusses on safety of a system as a whole rather than individual components, provided the ability to map algorithmic systems within their sociotechnical context. It focussed on three hazards, which are system states that lead to accidents. These three hazards were the use of flawed logic in algorithmic systems, the use of flawed information in algorithmic system, and the flawed understanding citizens have of the role of the systems. They were explored through a Safety Control Structure, which displays controlled processes and those controlling them, and the construction of causal scenarios. Semi-structured interviews focussed on the governance of algorithmic systems, and possible improvements. They provided empirical input for STPA. The main contributions of this research are the proposal it makes for a sociotechnical description of algorithmic systems and their harms, using concepts derived from STPA. Such descriptions can be used to build shared understanding of algorithmic systems and their harms. It showed how such a description can be constructed using an Actor Analysis, interviews, and STPA, describing how these methods work together.

The Dutch electricity network is shifting from a centralized, unidirectional system to a decentralized, bidirectional one, driven by renewable energy integration and increased electrification. This transition poses challenges, particularly low-voltage grid congestion. The study investigates the role of smart appliances in mitigating congestion through demand response initiatives. It identifies regulatory uncertainty, insufficient financial incentives, and lack of standardized communication protocols as barriers to adoption. Using qualitative interviews and Critical Transactions Theory, the research recommends regulatory intervention, direct load control, and unified communication standards to enhance grid resilience. Future research should incorporate social factors and assess standards for smart appliances.

The Dutch electricity network is shifting from a centralized, unidirectional system to a decentralized, bidirectional one, driven by renewable energy integration and increased electrification. This transition poses challenges, particularly low-voltage grid congestion. The study investigates the role of smart appliances in mitigating congestion through demand response initiatives. It identifies regulatory uncertainty, insufficient financial incentives, and lack of standardized communication protocols as barriers to adoption. Using qualitative interviews and Critical Transactions Theory, the research recommends regulatory intervention, direct load control, and unified communication standards to enhance grid resilience. Future research should incorporate social factors and assess standards for smart appliances.

The Dutch electricity network is shifting from a centralized, unidirectional system to a decentralized, bidirectional one, driven by renewable energy integration and increased electrification. This transition poses challenges, particularly low-voltage grid congestion. The study investigates the role of smart appliances in mitigating congestion through demand response initiatives. It identifies regulatory uncertainty, insufficient financial incentives, and lack of standardized communication protocols as barriers to adoption. Using qualitative interviews and Critical Transactions Theory, the research recommends regulatory intervention, direct load control, and unified communication standards to enhance grid resilience. Future research should incorporate social factors and assess standards for smart appliances.

This thesis investigates the integration of Large Language Models (LLMs) within large organizations operating in low resource language (LRL) regions, with a particular focus on Íslandsbanki, one of the prominent banks in Iceland currently grappling with the adoption of LLM technology. The research aims to understand the frameworks governing LLM integration within the bank, explore stakeholder perspectives on the integration process, and identify blind spots and gaps in their understanding of associated risks in order to understand what a comprehensive approach would look like. The methodology had mixed methods. It consisted of a comprehensive literature review followed by interviews with stakeholders involved in the LLM integration at Íslandsbanki. Subsequently, an Impact-Probability Assessment survey is conducted, where stakeholders are asked to evaluate the identified risks in terms of their potential impact and probability. This approach facilitates the identification and prioritization of risks, thereby enabling future integrators to navigate the integration process more effectively. Furthermore, data analysis techniques were applied to the survey results to provide quantitative insights into the identified gaps and perspectives. Findings reveal that perspectives on risks associated with LLM integration vary significantly, particularly between individuals with technical backgrounds and decision-makers within the bank. Moreover, the concept of text inflation emerges as a novel risk factor, highlighting the importance of considering diverse perspectives in risk assessment and mitigation strategies. Emphasis is placed on adopting a sociotechnical perspective to comprehensively assess risks at an organizational level. This research contributes to the understanding of the organizational impact of LLM integration, offers insights into diverse risk perspectives, and underscores the importance of considering sociotechnical factors in the integration process. By shedding light on the challenges and opportunities associated with LLM integration in the financial sector, this study aims to inform future research and practice in this evolving domain.

The thesis explores the complexities and risks associated with Automated Decision-Making (ADM) systems in social welfare, focusing on the phenomenon of ”digital cages.” The digital cage concept refers to a situation where rigid systems, perpetuated through algorithmic misclassification, inadvertently trap citizens in bureaucratic complications without recourse. This thesis analyzes the digital cages concept from a system safety perspective, on the example of the Dutch Toeslagenaffair. The thesis centers on understanding how digital cages form within social welfare systems and seeks methods to mitigate their emergence through targeted interventions using system safety theory. The pivotal question addressed is: ”What are safety control interventions to curb the emergence of algorithmic decision-making systems-induced digital cages in the context of national social welfare administration in the Netherlands?” The research concludes by highlighting the potential of assumption-based leading indicators. Assumptionbased leading indicators are predictive measures that rely on underlying assumptions about how certain inputs or actions correlate with future outcomes. The leading indicators provide a novel approach to enhancing system safety in social welfare administration by addressing the underlying assumptions of the system in operation. This offers a substantial potential to transform how social welfare systems manage and utilize ADM, potentially reducing the incidence of digital cages. By detecting early signs of hazardous systems states digital cages can be mitigated before they solidify into systemic issues. The thesis not only addresses the immediate concerns regarding digital cages but also opens pathways for future inquiries into safer and more equitable administrative practices. Future research is suggested to further refine these indicators and explore their applicability in diverse administrative contexts, ensuring they can effectively adapt to the evolving landscape of public administration and technology.