Large language models (LLMs) are increasingly deployed in consequential settings, yet their failures remain challenging to understand. Unlike traditional software bugs, such undesirable behavior emerges from distributed, context-dependent interactions that resist straightforward debugging. While XAI methods can surface signals about individual predictions, they do not directly support the hypothesis-driven investigative process that characterizes diagnosis in practice: forming expectations, gathering evidence, and identifying recurring failure patterns.

This thesis addresses this gap by introducing (1) a diagnostic framework that structures diagnosis through symptoms (observed undesirable outputs), signs (evidence from interpretability methods), and failure patterns (recurring, explainable combinations of symptoms and signs), complemented by a Should-Know/Really-Know lens that distinguishes task expectation from actual model knowledge; and (2) a prototype diagnostic toolkit that operationalizes this framework through integrated evaluation, run comparison, and state externalization.

An evaluation study with eight practitioners using codebook thematic analysis reveals three core findings. First, practitioners universally adopt a baseline-first strategy, building diagnostic confidence through initial evaluation before deeper probing. Second, they triangulate across samples, metrics, and interpretability outputs rather than relying on single signals, using comparison as a central sense-making operation for hypothesis testing. Third, diagnostic depth is systematically gated by three factors: interpretation friction (insufficient guidance on what methods reveal and how to act on their outputs), missing workflow glue (the absence of affordances for iterative refinement), and execution constraints (opaque platform limits that disrupt sustained diagnostic progress).

These findings reframe diagnostic tooling as infrastructure for iterative, hypothesis-driven reasoning, extending beyond the provision of isolated analytical methods. Effective diagnostic support must scaffold the full investigative cycle: from expectation formation and baseline calibration through evidence triangulation, hypothesis testing via comparison, and state externalization. This scaffolding must also account for the gating factors that shape the depth of diagnostic progress. This positions diagnosis as a knowledge and workflow challenge, with implications for tooling design, framework development, and empirical research into practitioner diagnostic workflows.

Applying Large Language Models (LLMs) to high-stakes classification tasks like systematic review screening is challenged by prompt sensitivity and a lack of transparency. We introduce IMAPR (Iterative Multi-signal Adaptive Prompt Refinement), a novel framework where a single LLM uses its own internal signals to iteratively refine its prompts, improving classification robustness and reliability. Unlike black-box optimizers that tune the prompts using only external scores, IMAPR is a white-box approach that diagnoses why a prediction failed using three internal signals: model confidence, a rationale, and a knowledge alignment score that checks whether the evidence cited in the rationale actually covers the user-defined inclusion criteria. We evaluate IMAPR on a real-world biomedical screening task, comparing it against strong baselines including GPO and StraGo. IMAPR outperforms the best baseline (GPO) by 8.8% in Macro-F1 while maintaining high, stable recall across runs. Across seven LLMs, IMAPR yields an average 9.2% improvement in Macro-F1 An ablation shows that knowledge-alignment acts as a recall safeguard: removing it leaves Macro-F1 similar but degrades recall, reducing reliability for screening. These results suggest that diagnostic, signal-driven prompt refinement is a practical alternative to black-box optimization for transparent, dependable LLM screening systems.

The use of research assistants has increased significantly, providing support and automation for researchers. However, there is limited research on researchers using research assistants and what assistance researchers require for each research stage.
We interview researchers to gather insights into their opinions and usage, and afterwards develop a prototype. Researchers highlight that the most difficult part of research is the experiment design, which is reflected in the lack of literature.
Participants use research assistants for reading literature and writing, but request more support. The research assistant must be factual, transparent, and correct, and including a conversation allows for feedback and discussion.
We evaluate the prototype for the experiment design phase, highlighting the effectiveness of the component architecture by generating correct experiments.

Large language models (LLMs) are widely used tools that assist us by answering various questions. Humans implicitly use contrast as a natural way to think about and seek explanations (i.e., "Why A and not B?"). Explainability is a challenging aspect of LLMs, as we do not truly understand how good the LLM answers are. The challenge is understanding to what extent LLMs can generate effective contrastive self-explanations for users. We introduce the Contrastive Self-Explanation Method (CoSEM) to narrow the gap between LLMs and explainability. It generates contrastive self- explanations and evaluates them through automation and a user study on generality, usefulness, readability, and relevance. Our results indicate that LLMs are capable of generating effective contrastive self-explanations. Lexical analysis of contrastive explanation indicates that explanations are not less general than the text those explain, and semantic analysis shows that more complex models generalize self-explanations more consistently. Although it is challenging to evaluate contrast in self-explanations semantically, user study shows that some models (Llama3-8B) help understand the contrast. Moreover, task selection affects how readable users find the explanations, where more self-explanations on general topics (movie reviews) are more readable than more specific topics (medical diagnoses). Lastly, some models, such as Llama3-8B, excel at generating contrastive self-explanations that contain relevant information regarding input text.

User feedback plays a significant role in helping recommendation systems to make personalized and accurate predictions. Despite the fact that many methods of collecting user feedback have been proposed, little research exists that addresses both the breadth and depth of data collected. In this study, we incorporate personal reflection into traditional crowdsourcing tasks and investigate how it facilitates people's reflection on their usage of YouTube. We present a novel crowdsourcing approach with personal reflection integration based on several design principles, which allows participants to reflect on contextual factors and personal values of using YouTube through guided context recall and evaluations, therefore gathering deeper insights into people's preferences and behaviors on a large scale. We conducted a user study involving 20 participants and explored the insights generated from their textual answers and the role of design principles in the reflection process. This approach successfully enables multiple participants to conduct the study simultaneously, thereby reflecting on their watching behaviors and preferences. Based on the quantitative and qualitative analysis of the findings, we sum up the implications of this approach to provide guidance for the YouTube recommendation system and point to the directions for the design of similar studies in the future.

This thesis investigates the enhancement of sentence decomposition in Large Language Models (LLMs) through the integration of linguistic features, including constituency parsing, dependency parsing, and abstract meaning representation. Traditional decomposition methods, which often rely on rule-based approaches, struggle with highly intricate sentences. By incorporating detailed linguistic features and employing reasoning steps and supervision in prompts, we aim to improve the comprehension and decomposition capabilities of LLMs. Experimental results show that integrating these linguistic features has the potential to improve decomposition performance. However, these enhancements also introduce new error types and increase computational costs. This study identifies several error types in the programs generated by LLMs, including format, decomposition, and conversion errors, emphasizing the need for further refinement in model training and prompt design. This research lays a step toward more accurate and efficient processing of complex sentences in LLMs, encouraging ongoing development and optimization in this field.

With the advent of large language models (LLMs), developing solutions for Natural Language Processing (NLP) tasks has become more approachable. However, these models are opaque, which presents several challenges, such as prompt engineering, quality assessment, and error analysis. Explainability methods can have several potential benefits, such as improving accuracy, increasing trust, and assessing quality. However, limited research exists on how explainability techniques can be applied to LLMs in practice, particularly using human-centred methodologies. Therefore, this study takes a user-centered approach, investigating the needs and challenges of the NLP data scientist and developing an explainability tool to address these needs. This approach is done by conducting a formative study to deepen our understanding of the user, combined with relevant literature. The observations from the formative study were used to develop a tool tailored to the user’s specific needs. This development was done by creating requirements and a design based on the findings of the formative study, followed by a proof of concept implementation. User satisfaction was assessed through practical interviews with a fairness dataset, providing insights into the usefulness and usability of the explanation techniques and the tool. The tool implements three explanation techniques: uncertainty, token-level feature attribution, and contrastive explanations. These can be viewed using a web application separated from the Python development environment, making it easy to interact with. Other key features are that it can be easily integrated into the user’s existing workflow, is usable in practice and can be presented to different stakeholders within the project. The evaluation concluded that the tool fits the workflow and does indeed help the NLP data scientist to understand the model. However, the evaluation also showed that the explainability techniques did not provide the necessary insights to achieve the user’s goal, mainly to improve the model’s accuracy and make the error analysis actionable. More research should be done to see which other explainability techniques could provide insights that would lead to objectively better performance of these models. Finally, more explainability techniques should be developed that do not focus on debugging the model but rather on revealing its behaviour and thus providing a better understanding of how to improve it.

Summarising patient interactions creates a huge workload for the healthcare professionals. This research finds that patient interactions contain a lot of noise that is subjective of nature. To explore the problem area interviews with a summarisation prototype have been conducted to extract system requirements and validate those by implementing them in the prototype. Filtering noise, reinforcement learning and numerical factual correctness are of paramount importance to a successful summarisation system.

With the rapid development of Artificial Intelligence (AI), the size and complexity of models are increasing rapidly. The limited memory and computing power of microcontroller units (MCUs) pose significant challenges for running AI applications on them. This thesis presents a method to run deep learning models on MCUs using a distributed approach.

First, we identified memory size as the primary constraint for implementing deep learning models in MCUs. To address this issue, our method involves splitting the models into smaller weight fragments distributed across multiple networked MCUs. A coordinator MCU manages the overall process, including neuron mapping and data relaying. We demonstrated that our approach reduces peak RAM usage during inference compared to existing methods.

For optimization, we employed layer fusion and quantization to reduce model size while preserving accuracy. We also introduced a rating system to assign capability scores to MCUs for efficient task allocation, explained through mathematical equations.

Our simulation phase validated the effectiveness of our method, showing successful distributed inference in MCUs and providing insights for real-world applications. Implementation on a network of MCUs confirmed the practical applicability and efficiency of our approach.

In conclusion, this thesis presents a feasible and efficient distributed inference method for networked MCUs, addressing resource limitations and enabling practical AI applications on constrained MCU platforms.

In the digital age, the proliferation of personal data within databases has made them prime targets for cyberattacks. As the volume of data increases, so does the frequency and sophistication of these attacks. This thesis investigates database security threats by deploying open source database honeypots to gather threat intelligence. We utilized five different honeypots at various interaction levels, deploying a total of 275 low-interaction, 50 medium-interaction, and 8 high-interaction honeypots over 20 to 23 days to collect a wide range of adversarial data. Through this deployment, we gathered 37, 618, 111 log entries from 8, 786 IPs.
Our analysis of these logs indicate that databases exposed to the internet are most likely to be dis-covered within an hour of deployment due to pervasive internet scanning. Additionally, we found that adversaries exhibit preferences for attacking certain database management systems, engage in irregular attack frequencies marked by short bursts, utilize diverse tools, and exploit both cloud service providers and infected machines. The findings also provide a high-level overview and analysis of observed attacks, including remote code execution, worms, botnets, data theft, and cryptojacking.

Large language models (LMs) are increasingly used in critical tasks, making it important that these models can be trusted. The confidence an LM assigns to its prediction is often used to indicate how much trust can be placed in that prediction. However, a high confidence can be incorrectly trusted if it turns out to be incorrect, also known as a high-confident error, or unknown unknown (UU). Blind spots, clusters of UUs, are caused by out-of-distribution data (OOD), bias, covariate shift, or unseen data. Previous work by Lippmann et al. generated samples based on these UUs, however, they selected a random subset of all found UUs. Although this was the first study that could mitigate blind spots without the need for crowd workers or oracles, they did not address the underlying causes of blind spots, such as OOD and bias. Data valuation is the research field that is concerned with valuing samples to improve model performance and can identify OOD data, bias, and noise. This paper proposed to combine the research field of data valuation, with the mitigation of blind spots. We generated synthetic samples using the highest-valued samples and retrained the LM using a Weighted Loss based on Data Values (WLDV). We conducted an extensive evaluation of our approach on four tasks, demonstrating a reduction of UUs by up to 32.7%, while retaining the same level of accuracy. This was the first exploration of combining the research fields of data valuation and blind spot mitigation.

In recent years, there has been a growing interest among researchers in the explainability, fairness, and robustness of Computer Vision models. While studies have explored the usability of these models for end users, limited research has delved into the challenges and requirements faced by researchers investigating these requirements. This study addresses this gap through a mixed-method approach, involving 20 semi-structured interviews with researchers and a comprehensive literature analysis.
Through this investigation, we have identified a practical need for a data model that encompasses the essential information researchers require to enhance explainability, fairness, and robustness in Computer Vision applications. We developed a data model that holds the potential to improve transparency and reproducibility within this field, speed up the research process, and facilitate comprehensive evaluations, whether quantitative or qualitative, of proposed methodologies. To refine and demonstrate the practicality of the data model, we have populated it with four existing datasets. Additionally, we have conducted two user studies to validate the model's usability. We found that participants were enthusiastic about using the data model. Some potential uses described by the participants were comparing models and datasets, accessing (niche) datasets and models, creating and exploring datasets, and having access to ground truth explanations. However, participants also had concerns about the data model, mainly with its usability being restricted to people with database knowledge and the richness of data in the database. Nonetheless, hope that this research constitutes the first step for data modelling for researchers in the field of Trustworthy AI.

Despite the low adoption rates of artificial intelligence (AI) in respiratory medicine, its potential to improve patient outcomes is substantial. To facilitate the integration of AI systems into the clinical setting, it is essential to prioritise the development of explainable AI (XAI) solutions that improve the understanding of the AI predictions. These XAI solutions empower clinicians to collaborate effectively with AI systems, thereby enhancing the overall outcomes for patients in respiratory medicine. Unfortunately, the lack of user-centric studies in this domain has made it challenging to identify the specific aspects of explainability that are most effective in improving the adoption of AI in the real-world environment. To address this gap, we conducted a mixed-methods study of clinicians in respiratory medicine to identify the most relevant and crucial aspects of XAI solutions. Our study focused on understanding how XAI can be effectively translated into clinical practice by leveraging the expertise of doctors in the field. Because of the lack of knowledge about XAI concepts among pulmonologists a different approach is taken to regular user-centric XAI research and no direct examples of state-of-the-art XAI solutions are used. Rather the expertise of doctors is used to make them implicitly identify their needs and intents. Our findings reveal that the successful adoption of XAI solutions in respiratory medicine requires tailored solutions that address communication barriers, promote patient-centric care, and overcome AI adoption challenges. The study highlights the significance of task-specific visualisations, comprehensive explanations, preferred granularity, and the ability to mimic human judgement in successful XAI solutions. Trust and collaboration between clinicians and AI systems are essential for effective adoption, wherein AI is perceived as a colleague rather than a replacement. This ensures that clinicians can easily understand and work with the model predictions, ultimately leading to improved patient outcomes. By aligning XAI design with the needs and intents of pulmonologists, we established the importance of Co-designing solutions with domain experts and embedding XAI within clinical workflows emerged as key strategies. Our research underscores the imperative of transparency, extended validation, and continuous alignment of AI technologies with medical values. By following these principles, XAI solutions can be developed to enhance the diagnosis and treatment of respiratory illnesses, ultimately improving patient outcomes in respiratory medicine.

A 2022 Harvard Business Review report critically examines the readiness of AI for real-world decision-making. The report cited several incidents, like an experimental healthcare chatbot suggesting a mock patient commit suicide in response to their distress or when a self-driving car experiment was called off after it resulted in the death of a pedestrian.

These incidents, leading to media frenzies and public outcries, underscore a pressing concern: "How do these AI systems reach their conclusions?" It has created an urgent demand for transparency and clarity in AI decision-making processes. This urge to understand has translated into a significant uptick in the volume of work in Explainable AI (XAI). This makes it crucial to have consistent evaluation standards for streamlined growth in the field.

However, XAI, being a multidisciplinary field, faces the challenge of a lack of consensus on what constitutes a "good" explanation. Stakeholders with diverse backgrounds and needs can have diverging expectations from XAI. Some might prioritize simple and concise explanations, while others prioritize detailed information about AI predictions, all depending on their end goal.

This thesis addresses the standardization of an evaluation framework for XAI methods, that accounts for stakeholders' needs in different usage contexts. It presents a prototype that can be customized and extended to suit various XAI methods and tasks. Findings affirm the framework’s ability to yield insightful comparisons between different XAI methods. It also highlights issues with human perception of specific XAI features in those methods. The efforts in this work contribute to XAI techniques being integrated into real-world applications, ensuring more reliable and consistent performance assessment.

Colorectal cancer is a widespread disease that significantly impacts the health of individuals worldwide. Understanding the needs and concerns of those affected by this disease is crucial for improving patient outcomes and enhancing the quality of care. Patient web forums have emerged as valuable platforms for individuals to openly share their experiences and thoughts related to colorectal cancer, providing unique insights into the social, physical and emotional aspects of their patient journey. These forums offer a more comprehensive and authentic portrayal of patient experiences compared to traditional patient data collection methods, such as questionnaires and interviews, which may not capture the full scope of patients experiences in the colorectal cancer carepath.

However, analyzing the vast amount of unstructured data within these patient web forums presents a significant challenge. Traditional manual analysis by human experts is time-consuming, labor-intensive, and limited in scalability, making it impractical to analyze the sheer volume of patient-generated content. This is where the application of natural language processing (NLP) techniques becomes crucial. NLP enables the automated processing and analysis of textual data, allowing for efficient extraction and interpretation of the large amounts of patient forum posts.

Nevertheless, relying solely on machine intelligence, such as topic modeling and natural language generation, for interpreting patient forum data carries inherent risks, including the potential for disseminating misleading information. While these machine-driven techniques offer efficient and scalable ways to analyze and generate insights from the large amount of diverse and unstructured patient forums, they may lack the necessary contextual understanding and domain expertise to ensure the accuracy, relevance, and ethical implications for interpreting colorectal cancer patient experiences.

To close this gap between human experts and machine intelligence, this thesis explores the potential of hybrid intelligence (HI) for topic interpretation in colorectal cancer research. The main research question is: ``How can topic modeling, GPT-3.5 language generation and human expertise be combined to explore the interpretation of patient web forums in colorectal cancer (CRC) research?"

To address the research question, three human studies were conducted. The first study employed NMF topic modeling to compare topic interpretations created independently by medical workers and GPT-3.5. This comparative analysis discovered unique observations that differentiate human-written and AI-generated interpretations on online patient stories. In the second study, it was investigated how medical researchers collaborate with GPT-3.5 to develop hybrid interpretations on patient experience topics generated by the BERTopic model. A Flask web application served as the interactive platform for combining their knowledge with the AI model. Finally, the third study made professional human evaluators assess the topic relevance of the interpretations generated by medical researchers and GPT-3.5 to determine whether the combination of GPT-3.5 and human expertise leads to improved topic interpretations compared to individual interpretations.

The proposed solution to the research problem is to explore a hybrid workflow that compares, combines and validates GPT-3.5 language generation and human expertise, aiming for enhanced interpretations of topics extracted from colorectal cancer patient forums. The three studies provide opportunities for researchers and medical professionals to integrate machine intelligence from topic models and GPT-3.5 in their field of work. The hybrid workflow has conclusively demonstrated that human experts were successfully able to compare and enhance the relevance of human and GPT-3.5 interpretations of colorectal cancer patient experience topics. This allowed human experts to efficiently reach a more comprehensive understanding of patient forum data, which is essential for improving patient health in colorectal cancer research.

Child helplines play a crucial role in delivering expert assistance to young clients facing challenges and seeking support. While counselling is instrumental in enhancing children’s mental well-being, the limited number of experienced counsellors is inadequate given the substantial workload. At this point, effective training to volunteer counsellors becomes essential. This thesis aims to expand upon a Belief-Desire-Intention (BDI) based chatbot, the simulated child, with the specific objective of enhancing volunteer training at child helpline organizations through the integration of emotional capabilities. Through the collaboration with professionals from a Dutch child helpline, the Kindertelefoon, we identified the predominant emotion of frustration along with its underlying triggers. Armed with this insight, we proposed an Emotion-BDI model capable of dynamically adjusting the chatbot’s displayed emotion in response to its environment. The evaluation of a prototype constructed based on this model supported our hypotheses that a chatbot equipped to express frustration possesses greater believability and emotional presence compared to the original version. Moreover, it also indicated heightened user enjoyability, engagement, and perceived usefulness. Within this thesis, we highlighted the enhanced feasibility of the simulated child brought about by our proposed Emotion-BDI model.

Machine learning (ML) systems for computer vision applications are widely deployed in decision-making contexts, including high-stakes domains such as autonomous driving and medical diagnosis. While largely accelerating the decision-making process, those systems have been found to suffer from a severe issue of reliability, i.e., they can easily fail on serving data that are slightly different from the data captured during their training phase. Such an issue has resulted in undesired outcomes with safety, ethical, and societal concerns across various applications, such as numerous examples of semi-automatic cars causing accidents on the road.
In this thesis, we hence develop a system in order to support ML practitioners in debugging their computer vision models, even before deploying them and having access to serving data.

We take inspiration from prior ongoing works in order to formulate the current diagnosis problem, identify its challenges, and envision a human-computation-based solution. We then thoroughly analyse the requirements for developing a system instantiating the solution, actually design such a system, and implement it in a well-functioning, full-fledged, highly-modular, and easily-customizable system.
The solution is based on the definition of human computation operations, that, altogether, allow to a) identify the mechanisms a human would expect the model to learn in an ideal world, b) identify the mechanisms the model has actually learned (via annotations of saliency maps), and c) to compare these two sets of mechanisms to conclude about the good behavior of the model. The solution is especially made to account for certainty issues in the work of the human workers, and to handle ambiguous granularities in the concepts the model might have learned.
To the best of our knowledge, our work is the first system that allows an ML practitioner to first identify their own goals for debugging a model (among a large diversity of goals), accounting for their limited monetary budget, then to configure a debugging session according to these goals, and finally to fully-automatically run the system with such configuration to obtain a model debugging report.

Finally, we conduct a thorough investigation of our system. First, we set-up to understand the correctness and informativeness of the outputs, by running the system with various configurations on different models trained using various datasets, for which the biases are more or less controlled. This first evaluation particularly shows that the outputs and its implementation are correct. With these outputs, we are able to identify the biases that have been injected in the model, as well as to learn about previously unknown behaviors of highly-common models that are used by many practitioners.
Second, we evaluate the cost-effectiveness of running the system. For that, we ran tests in two settings: when the human workers might make mistakes (e.g., due to a lack of expertise, the complexity of the task, or inattention), and when human workers are fully accurate. We vary the configurations of the system (e.g., the order in which the human operations are conducted, the number of workers allocated at the start of the debugging session) within the two settings, and we observe how the number of human operations needed evolve, in order to reach correct system outputs. We find that the system's output is potentially relevant, informative and complete. The system output provide an in depth analysis of the model's behaviour and unravel what the model comprehends, where it falls short, and what it should ideally have grasped.

All in all, in this thesis, we build the system and thoroughly evaluate it. While we identify a number of conceptual and practical limitations of this system (e.g., difficulty to annotate concepts, potentially high cost), our work constitutes a first step towards developing complete solutions to help practitioners debug their system. We encourage readers to build on our work, in order to further optimize our system for cost. Note that we make all our code publicly available for anyone to re-use our system, or reproduce our experiments.

In contrast to the prevalent focus on real photos in computer vision research, we present a contribution by making the Ot & Sien dataset machine learning-ready for object detection tasks in illustrations. We refer to the new dataset as Ot & Sien++ that is composed of scanned images of children’s book illustrations, thereby venturing into an unexplored domain. The primary objective of this research is to investigate the generalization capabilities of existing object detection models to this unique dataset and establish benchmarks for this dataset.
To evaluate the performance of existing object detection models on our proposed dataset, we employed the widely used YOLOv5 as a benchmark. To mitigate the inherent imbalance of the dataset, various data augmentation techniques were applied. The results demonstrated the effectiveness of the object detection model and data augmentation in the context of children’s book illustrations. In addition, this research also explored applying few-shot learning models to the dataset. Baseline models were investigated to examine the potential of few-shot learning in the context of object detection in illustrations.
The proposed dataset elicits new challenges in object detection and will serve as a valuable resource for researchers in this domain. Our dataset can be found at https://data.4tu.nl/datasets/d1f3ca5c-f1e4-48f5-9a04-0564572d2b9c/1.

Language is an intuitive and effective way for humans to communicate. Large Language Models (LLMs) can interpret and respond well to language. However, their use in deep reinforcement learning is limited as they are sample inefficient. State-of-the-art deep reinforcement learning algorithms are more sample efficient but cannot understand language well. This research aims to study whether RL agents can improve learning by utilizing language assistance and how LLMs can help them. A sentence describing the agent's environment is fed into an LLM to create a semantic embedding, which is consumed by a recurrent Soft Actor-Critic (SAC) agent to create an agent that can listen to natural language. This research shows that the best method for the agent to consume the embedding is concatenating it to each observation. Also, LLM-based embeddings lead to faster and more stable learning than non-LLM-based embeddings. The agent is sensitive to noise in the embedding but not to the embedding's dimensionality. The agent can generalize well across sentences that have a similar meaning to sentences seen during training but are formulated differently, but it can not generalize as well across sentences with unknown subjects and needs the subjects of the sentences to be grounded in training. Lastly, this research shows that the proposed architecture supports scaling language assistance to more complex environments.