Reinforcement learning (RL) agents often achieve impressive results in simulation but can fail catastrophically when facing small deviations at deployment time. In this work, we examine the brittleness of Proximal Policy Optimization (PPO) agents when subjected to test-time observation noise and evaluate techniques for improving robustness. We compare four variants—feed-forward PPO, Recurrent PPO (with LSTM memory), Noisy-PPO (trained with injected observation noise), and Recurrent-Noisy PPO—across two benchmarks: the classic CartPole-v1 and the more realistic Highway-env. Performance is measured over 100 episodes per corruption level, using mean return, success rate, and the Area-Under-Degradation-Curve (AUDC) as robustness metrics. Our results show that noise-augmented training yields the largest gains, with Noisy-PPO maintaining its clean-condition performance even at high noise levels, while recurrence alone offers more modest improvement. In the Highway-env, both noise injection and LSTM memory improve returns, indicating that a simple integration of noise augmentation or recurrence can enhance PPO’s robustness to real-world uncertainties.

Reinforcement learning agents are trained in well-defined environments and evaluated under the assumption that the test time conditions match those encountered during training. However, even small changes in the environment’s dynamics can degrade the policy’s performance, even more so in safety critical domains. This work investigates the use of Quantile Regression Deep-Q Networks (QR-DQN) to detect environment shifts by analyzing the uncertainty in return predictions. QR-DQN extends Deep-Q learning by estimating the entire distribution of future returns through quantile regression. We hypothesize that under deterministic settings, the spread of the return distribution, quantified by the inter-quantile range, can determine whether environmental changes took place. The RL agent learns low spread predictions for familiar dynamics, but when deployed in changed environments, the quantile distribution becomes wider. We conduct experiments on the deterministic CartPole-v1 environment by the pole length. We show that the quantile spread is low under small changes, but drastically increases as the dynamics’ shifts diverge more from the training setting. Our results indicate the potential of distributional reinforcement learning to enhance reliability and awareness in deployment scenarios.

The Sim2Real gap poses significant challenges for testing autonomous vehicles, often becoming apparent only during high-risk real-world deployments. This research proposes a novel pipeline that leverages both high-fidelity (CARLA) and low-fidelity (Gym-Duckietown) simulators to estimate this gap prior to deployment. The results reveal a strong corelation between performance in Gym Duckietown and real-world outcomes, suggesting it can serve as potential estimation for real world performance and the Sim2Real gap. Nonetheless, real-world testing remains an essential part of the validation process. Future work should build on these findings to further explore and validate the approach.

Domain randomization (or DR) is a widely used technique in reinforcement learning to improve robustness and enable sim-to-real transfer. While prior work has focused extensively on DR in combination with algorithms such as PPO and SAC, its effects on value-based methods like DQN and QR-DQN remain underexplored. This paper investigates how varying degrees and types of DR affect the robustness and generalization capabilities of agents trained with DQN and QR-DQN. We identify clear differences in how DQN and QR-DQN respond to domain randomization, suggesting that naive application may hinder performance, whereas well-targeted distributions can enhance robustness and generalization. These findings underscore the importance of tailored DR strategies for different algorithms and contribute to a deeper understanding of DR’s role in DQN-based methods.

As autonomous driving systems advance, ensuring the robustness of underlying decision-making algorithms becomes increasingly critical. This study assesses the performance and reliability of two reinforcement learning models, Deep Q-Network (DQN) and Quantile Regression DQN (QR-DQN), within the context of a simulated highway environment. While DQN has been widely adopted for its simplicity and effectiveness in discrete action spaces, it suffers from overestimation bias and lack of performance in out-of-distribution environments. QR-DQN addresses some of these limitations by modeling the distribution over returns using quantile regression, offering a superior representation of uncertainty. This research focuses on two core objectives: (1) implementing a riskaverse decision-making strategy using the quantiles of QR-DQN to enhance safety and reliability, and (2) evaluating the robustness of DQN and QR-DQN as the test environment deviates from training conditions. Results show the limitations of DQN and demonstrate QR-DQN’s higher robustness in different environments. Moreover, a better performing alternative of QR-DQN is presented, employing a conservative behaviour through the use of its quantiles. This puts emphasis on the implemented model’s trade-off between maximising rewards and avoiding collisions, providing a safer approach.

This thesis utilizes Evolutionary Algorithms (EAs) within the BRIGHT framework for developing breast cancer brachytherapy treatment plans. We use expert knowledge and state-of-the-art EAs to formulate treatment planning as a multi-objective optimization problem whose solutions can be applied to actual patients. We propose four novel 2- and 3-objective formulations of this problem, which we implement within BRIGHT and empirically validate against anonymized data from 9 real-world patient cases. We demonstrate that all four formulations, under reasonable computational and time budgets, are capable of generating plans that match or exceed the properties of reference treatment plans. To verify the clinical relevance of our contributions, we rely on the expertise of a clinical expert who assesses whether the generated plans can be used in clinical treatment planning. The results of our empirical analysis show that 17 of the 18 plans presented to the expert are clinically acceptable and of immediate value to practitioners within the field of breast brachytherapy today.

LLM (Large Language Model) powered AI (Artificial Intelligence) assistants are a popular tool used by programmers, but what impact do they have? In this thesis we investigate two such tools designed by JetBrains: AI Assistant and FLCC (Full Line Code Completion).
We collected over 40 million actions - including editing, code executions, and typing - in the form of metric data spread out over 26 thousand users. With this data, we look at how user behavior changes when assisted by AI Assistant or FLCC.

Users spent more time in their IDEs (Integrated Development Environment) and typed more when assisted. In most cases, we see a decline in (manual) testing, or at best an equivalent level. And how do multi-programming language benchmarks reflect acceptance rates by users for these respective languages? There seems to be no real correlation between these benchmark results and what users accept in their generations, but the available benchmarks are also limited.

In the end, more research is required to put these results in perspective, for now, it is still up in the air whether the changes in user behavior are positive or negative. But relating this to existing works the impact seems mostly positive on users of AI Assistant and FLCC.

Software testing is a vital yet time consuming process during the development lifecycle, often causing engineers to limit its use in practice. In order to encourage active software testing, researchers have shown significant advances in automatic unit test case gener- ation with approaches such as search-based testing (i.e., EvoSuite) and large language models (i.e., ChatGPT). However, while the first suffers with exploring edge cases of the input space, the latter still suffers from hallucinations during code synthesis, limiting the use of both solutions. This research aims to overcome these limitations by utilizing the strengths of both techniques, which are effective test structure generation and program inference, respectively. In particular, the assertions of initial unit tests generated by EvoSuite are augmented using ChatGPT-4o, with the aim of improving the mutation score, and hence the overall test suite effectiveness. We evaluate our solution, called evoLLve’M, on a benchmark of 20 Java classes from the SourceForge110 Corpus and compare it to only using EvoSuite, which is considered the state-of-the-art ap- proach. Results show that evoLLve’M outperforms EvoSuite in 25% of the classes for mutation score, without negatively impacting other classes. It boosts the total number of killed mutations by 3%, achieving the most improvement for mutations types of increments and null returns, being 26.9% and 8.9%, respectively.

Deformable Image Registration (DIR) is a process in which the point-to-point correspondence between two or more medical images is estimated. This could allow spatial data to be transferred between these images, easing the work of practitioners in the field of radiation oncology. Many DIR approaches already exist. These are not yet applicable in all clinical settings, however, as DIR is a complex problem, partly because of the difficulty of evaluating image registration results. This is why often multiple objectives are considered in this evaluation. In this thesis, we combine two separate multi-objective DIR approaches into hybrids while evaluating their performances, with the aim to improve upon the individual approaches, thus reducing the gap toward clinical practice. In these hybrids, we also use an approach called Automated Landmark Detection which automatically detects corresponding landmarks between the source and target images.

The first DIR approach, the Digital Phantom, creates a biomechanical model of the patient and uses an Evolutionary Algorithm (EA) to optimize parameters of a Finite Element Method (FEM)-based simulation. The second DIR approach, MOREA, creates a dual-dynamic biomechanical transformation model of the patient, using an EA to optimize its parameters.

In the first hybrid approach, called the Landmark Guided Digital Phantom (LGDP), the Automated Landmark Detection approach is used to automatically find corresponding landmarks, which are then used as a guidance objective in the Digital Phantom. The second hybrid is called DP-MOREA, where the output of the Digital Phantom is used to initialize deformations in the MOREA approach. The final hybrid is called LGDP-MOREA. This hybrid combines all three components, using the generated landmarks to guide the Digital Phantom and initializing MOREA with the result. We analyze the effect of these hybrids on the quality of the final registration, both quantitatively and qualitatively. Additionally, we compare these hybrids with the components they are composed of.

We test the performance of the approaches and hybrids on pelvic CT scans of three cervical cancer patients with large deformations. We observe that the individual Digital Phantom component outperforms the LGDP hybrid both quantitatively and qualitatively on these three patients. In addition, we observe that initializing MOREA with the output of the Digital Phantom and LGDP improves on how fast the optimization converges, that it improved the end results quantitatively and that it also has an effect on the resulting Deformation Vector Fields (DVFs). We conclude that in future research, these effects on the DVFs and their clinical relevance should be studied in more detail.

Addressing the challenge of reasoning about programs across different evaluation strategies has long been a concern in functional programming. Levy's introduction of the call-by-push-value (CBPV) calculus represents a significant step forward in tackling this. His paradigm provided a more powerful approach that can encapsulate both call-by-value and call-by-name that was even later extended to include call-by-need. In this paper we present the development of an interface that integrates the theory of CBPV with algebraic effects and handlers. We demonstrate how this technique enables the definition and execution of programs, highlighting its capability to defer computations across different evaluation strategies and define operations in a modular fashion. We then define and prove a set of laws that can be used with our interface to reason about programs under varying evaluation regimes. This approach not only enhances the flexibility and modularity of language and library implementation but also allows for direct reasoning about these implementations, beyond the meta-level abstraction.

Algebraic effects and handlers are a new programming technique that allows for the definition of abstractions as interfaces, with handlers providing modular, concrete implementations of these interfaces. In this paper, we consider algebraic effects and handlers implemented in Haskell, and explore how they behave under fixed-point (value) recursion. We give different possible implementations of fixed-point combinators for effectful functions, and work out their evaluation processes. We find that these functions behave very predictably under normal fixed-point recursion, while value recursion seems to be a much harder problem. We discuss the difficulties of implementing value recursion, and several possibile solutions are explored, but the question of whether a fixed-point combinator with value recursion semantics can exist at all in the presence of algebraic effects remains unanswered.

Algebraic effects and handlers are a new paradigm in functional programming. They aim at modularly handling side effects, by separating the declaration of those effects, from how they are handled. In this paper, we show how we can leverage their use to create an interface for concurrency using algebraic effects for which we can prove a list of concurrency laws, and also design handlers that allow us to run programs concurrently, thereby demonstrating the practical application of algebraic effects and handlers in managing concurrency.

Algebraic effects and handlers has been a popular approach for modelling side-effects in functional programming languages. Focusing on composability and modularity, this approach separates the effectful syntax from its semantics, which helps programmers to create effect abstractions such that their implementation can be modified without changing the syntax. However, there exist mainstream functional programming languages, like Haskell, which lack built-in frameworks to accommodate specifying side-effects in this manner. In this paper, we provide an interface for Haskell's Software Transactional Memory (STM), a concurrency abstraction, in the framework of algebraic effects and handlers from prior literature. We embed our implementation into a simple concurrency model using higher-order effects, in order to demonstrate it is possible to define and execute effectful concurrent programs that obey the semantics of Haskell's STM. Furthermore, we prove that our implementation satisfies the necessary laws governing our interface, such that programmers can easily reason about programs using our STM model.

Software development often relies on dependencies managed by package managers to simplify the integration of external libraries and frameworks, reducing development time. However, developers sometimes choose to bundle dependencies directly within their software packages. Bundling dependencies means including all necessary third-party frameworks directly within the application's distributable archive, such as a JAR file, to ensure all components are present without needing external installations. This practice, resulting in Uber JARs (or fat JARs), presents both challenges and advantages within the Maven ecosystem. This project examines the prevalence, risks, and impact of Uber JARs by analyzing over 9 million POM files and 12 million JAR artifacts from Maven Central, identifying artifacts with previously undetected vulnerabilities. Notably, 10.48% of the analyzed artifacts, amounting to 915,089, fall under the category of Uber JARs, indicating a significant prevalence within the Maven repository. Central to this work, JarSift detects Uber JARs' contents, including the libraries, their versions, and vulnerabilities. JarSift's accuracy is demonstrated with an F1 score ranging from 0.474 to 0.857, depending on the Uber JAR configuration. Analysis reveals about 17.13% Uber JARs in a small-scale dataset contained undisclosed vulnerabilities, and 0.63% of all libraries in our dataset fully completely matched known vulnerable libraries. These findings highlight the need for better detection and mitigation strategies in the Maven ecosystem and inform developers of potential risks, helping them implement more robust security measures.

Mixed-integer optimization problems, incorporating both discrete and continuous variables, present unique challenges across various domains such as computer science, finance, logistics, and healthcare. Evolutionary Algorithms (EAs) have emerged as powerful optimization techniques capable of tackling such complex problems in either the discrete or continuous domain. Model-based EAs, integrating machine learning techniques, have further improved the efficiency and scalability of these algorithms. The first algorithm to combine discrete and continuous model-based EAs is the Mixed-Integer Hybrid Evolutionary Algorithm (MIHEA). Despite its potential, MIHEA remains relatively underutilized in research endeavours. This thesis seeks to bridge this gap in research by applying the algorithm to a novel context: structure learning of Bayesian Networks (BNs).

BNs offer a transparent framework for probabilistic reasoning, making them well-suited for various applications. However, learning the structure and discretizations of BNs from data is a challenging task. It is not uncommon for the datasets to contain values of continuous nodes as well. Assuming no normality, these nodes in the data need to be discretized, since BNs are designed for discrete data. The optimal discretizations of these nodes depend on the structure of the BN, meaning that the discretization optimization should happen simultaneously with the structure learning. MIHEA holds promise for addressing this challenge by leveraging its mixed-integer optimization capabilities.

The investigation performed in this thesis starts with a reproduction study. The description of the code for MIHEA is shown to be inconsistent with the experiment results, which prompts a reproduction study resulting in a version of the code that more accurately fits the results.
Subsequently, MIHEA is applied to the structure learning of BNs, where discrete variables represent network structure and continuous variables encode discretizations of continuous nodes. Several solution representations are explored. The experiments show that MIHEA achieves similar or better performance than the state-of-the-art DBN-GOMEA approach on the task of recreating randomly generated BNs from data, at the cost of increased execution time.
These results demonstrate the potential of model-based mixed-integer EAs, particularly MIHEA, for BN structure learning from continuous data. The findings encourage further exploration and utilization of mixed-integer EAs in solving (real-world) problems involving BNs and continuous data.

n the rapidly evolving semiconductor industry, precise material characterization is crucial. This thesis focuses on enhancing Ion Beam Analysis (IBA), a pivotal tool in semiconductor characterization, through the integration of differential evolution optimization. This research proposes a single and multi spectra optimization algorithm approach controlled by a web application. Central to this work is a com- parative analysis of the proposed algorithms to simulated annealing and the DE algorithm proposed by Heller et al. [5]. This analysis shows a good performance of the proposed algorithms and a potential for industry application. The web application offers users a robust, user-friendly, and scalable interface for IBA optimization. By proposing and testing new IBA optimization methods, this thesis contributes significantly to semiconductor technology, offering new methods for material characterization at microscopic level.
This project is a joint project between Imec (Leuven, Belgium) and HZDR (Dresden, Germany).

Formally verified programs can be embedded in larger non-verified code bases by means of syntactically faithful source-to-source translation: systems like Agda2Hs make it possible to translate verified code written in a dependently typed programming language to a general-purpose functional programming language, Agda and Haskell in this case. Such systems can enable verification for critical functionality while keeping wider ecosystem access and easier to write code for peripheral functionality. However, this interfacing leaves a gap in the formal guarantees of the verified code: preconditions that were only expressible as a dependent type and have thus got erased upon translation might not be met. We present runtime checking these preconditions as a way to close this gap and ensure that computation does not continue on ill-formed input. As an extension to Agda2Hs, we have implemented a solution to automatically insert runtime checks for the preconditions and only make those definitions accessible in the output that are also checkable. The runtime check insertions do not only cover functions, but also data types and non-class records in the form of smart constructors; higher-order erased arguments are supported as well. We make a formal completeness analysis of a simplified version of the checks we generate for well- and ill-typed programs.
In our solution, class instances are not supported for runtime checking due to their different nature, and capabilities for recovering from a failed runtime check are still rudimentary. Despite these limitations, we conclude that a closure of the input precondition verification gap is possible, and that the development time trade-off in comparison to handwriting checks can be worthwhile.

Organizations use cloud service providers for outsourcing their data, since this includes advantages such as: scalability, security and no need for in house experts. Therefore, outsourcing data to cloud providers results in reduced costs.
The disadvantage of outsourcing data to a cloud provider, is that organizations are not in control of their own data.
When organizations are not in control of their data, they are subject to privacy risks.
Privacy risks should be avoided, especially when sensitive data such as medical or financial records are involved.
Therefore, organizations protect their data by only outsourcing encrypted data to cloud providers.
However, data analysis on encrypted data is significantly reduced due to computational and communicational overhead.

A commonly used data analysis method, such as k-Nearest Neighbour Search (k-
NNS), is useful when for finding similar records in a database for a given query.
Previous research shows success using k-NNS methods while preserving privacy, by using fully homomorphic encryption.
However, previous solutions required the client to be online and help in the protocol, or make use of non-colluding servers.

Therefore, we introduce our k-NNS protocol, which outsources all the work to the cloud server and the client is not involved in the computation.
Our k-NNS protocol shows success on data sets used to test k-NNS applications, however is significantly slower than solutions which involve the client or non-colluding servers.

Network Intrusion Detection Systems (NIDSs) defend our computer networks against malicious network attacks. Anomaly-based NIDSs use machine learning classifiers to categorise incoming traffic. Research has shown that classifiers are vulnerable to adversarial examples, perturbed inputs that lead the classifier into misclassifying the input. Existing work has shown weaknesses in classifiers for classifying network traffic, but none have shown the possibility of automatically recreating network attacks that can bypass existing anomaly-based NIDSs. Regular methods for generating black-box adversarial examples create packets that are invalid. We present AGONI, a Multi-Objective Genetic Algorithm for
generating network packets that are both valid packets and adversarial examples for NIDSs. AGONI can successfully generate valid adversarial examples in multiple attack scenarios. Against the NIDS Suricata, 99.93% of the generated adversarial examples can successfully bypass the defence. We compare the performance of AGONI against randomly generating network packets, the Boundary Attack and an adjusted version of the Boundary Attack which can better create valid adversarial examples. Only AGONI consistently generates valid adversarial examples when compared to the Random Attack (82%), the Boundary Attack (0%) and the Networking Boundary Attack (74%).