Scandals in which governmental ADM tools played a role, have recently brought about political and societal debate about the potential harms to citizens that such automated systems potentially bring. This thesis focuses on ADM systems which contain an AI component. Based on the knowledge gaps perceived, the main research question for this study is: In public sector AI systems, what are emerging vulnerabilities for citizens and how do these translate into governance requirements for decision-makers? The approach to answer this question is an adjusted form of Theory Building from Case Study. A layered ’onion model’ presents four relevant contexts to consider for AI System vulnerabilities specifically in the public sector: AI model, Model deployment, Political-administrative, and Societal. The case study results demonstrate that dealing with vulnerabilities in one of the four model contexts often complicates dealing with vulnerabilities in the other contexts. Hence, the vulnerabilities model points to so-called governance requirements dilemmas.