In today’s rapidly evolving software landscape, where continuous integration and continuous delivery are paramount, the presence of flaky tests poses a significant obstacle. These tests, exhibiting unpredictable pass/fail behavior, hinder development progress, waste valuable resources, and erode developer trust. This research delves into the root causes and mitigation strategies for flaky tests within a large-scale, database-driven industrial setting: Exact.

The increasing reliance on databases in modern software systems, including Exact’s own platform, necessitates a deeper understanding of the unique challenges posed by database-dependent tests. By analyzing flaky test behavior through repeated test runs on the same code, we identified key contributors to flakiness, including resource contention, test order dependencies, ‘dirty tests’ that leave the system in an inconsistent state, platform-specific issues, and combinations thereof.

Based on the root causes for flakiness at Exact, we developed and evaluated three mitigation strategies and supporting tools: minimizing redundant database background tasks, explicitly disposing of test data, and disabling database dirty tests. Our study resulted in a substantial reduction in flakiness, leading to a significant increase in the release rate from Exact from 60% to 96%. We improved the chance of their CI/CD pipeline passing with no code changes from 27% to 95%.

Furthermore, this research highlights the importance of collecting and analyzing rich, granular test data to identify patterns and root causes of flakiness. Providing developers with actionable information from this analysis motivates them to address flakiness proactively. Moreover, understanding the interplay between different types of tests, such as the impact of dirty tests on other seemingly unrelated tests or in combination with other factors, is crucial for effectively mitigating cascading failures.

WebDSL is a DSL for creating web applications, combining many different aspects and domains of web design in a single language. The dynamic semantics of this language are not defined, despite multiple attempts, abandoned due to complexity of the language and lack of expression of chosen frameworks. We adapt the algebraic effects and handlers approach and the framework introduced in Datatypes a la carte (Swierstra, 2008) to create a modular denotational semantics model of WebDSL, extending the framework by a bifunctor formulation for multi-sort syntax definition that allows us to distinguish between effects raised by different components of the language. In the process of defining the framework and semantics in Haskell, we encountered obstacles and of working with algebraic effects and handlers paradigm in the language, leading us to compile workarounds, solutions and pitfalls to avoid when constructing and maintaining such model. In evaluation of the framework approach with earlier attempts at defining dynamic semantics for WebDSL, we find algebraic effects and handlers to be a viable and successful approach for modelling a rich DSL such as WebDSL, and propose possible improvements to the WebDSL compiler.

Writing tests enhances quality, yet developers often deprioritize writing tests. Existing tools for automatic test generation face challenges in test under- standability. This is primarily due to the fact that these tools fail to consider the context, leading to the generation of identifiers, test names, and identifier data that are not contextually appropriate for the code they are testing. Current metrics for judging the understandability of unit tests are limited as they do not take into account contextual factors such as the quality of comments. Developing a metric to evaluate test readability is essential for selecting the most comprehensible tests. This research builds on UTGen, incorporating LLMs to enhance the readability of automatically generated unit tests. We developed a readability score and used LLMs to evaluate and rank tests, comparing these rankings with human evaluations. This research concludes that LLMs can successfully evaluate the readability of test cases. The GPT-4 Turbo Simple Prompt model exhibited the best performance, with a correlation of 0.7632 with human evaluations. Through comparing different LLMs and techniques for as- signing readability scores, we identified approaches that closely matched human evaluations, demonstrating that LLMs can successfully rate the read- ability of test cases.

Automated software testing is a frequently studied topic in specialized literature. Search-based software testing tools, like EvoSuite, can generate test suites using genetic algorithms without the developer’s input. Large Language Models (LLMs) have recently attracted significant attention in the software engineering domain for their potential to automate test generation. UTGen, a tool integrating LLMs with EvoSuite, produces more understandable tests than EvoSuite; however, the generated tests suffer a coverage drop.

To streamline bug detection by developers, we propose UTGenCov, a concept that focuses on improving the understandability of EvoSuite-generated tests without compromising on coverage. This approach builds upon UTGen by thoroughly analyzing the reasons behind the decrease in coverage and proposing an alternative approach.

Our investigation determined that the leading cause of coverage reduction in UTGen is LLM hallucination in the Understandability phase. UTGenCov aims to address hallucinations by providing the source code of the methods used in the test to the LLM. Yet, our experiment results indicate inconsistent performance and a further decrease in branch coverage of 0.74% compared to UTGen.

Automated test generation is the means to produce correct and usable code while maintaining an efficient and effective development process. UTGen is a tool that utilizes a Large Language Model (LLM) to improve the understandability of a test suite generated by a Search-Based Software Testing tool, namely EvoSuite. Often while the LLM attempts to improve a given test case, it generates code that is too far from the original, changing the test's purpose. Alternatively, it may generate code that does not compile. Such behaviour is called ``LLM Hallucination".

The current hallucination handling of UTGen is time-consuming and resource-expensive. To address this, we propose two alternative approaches that use information retrieval prompt engineering techniques to minimise hallucinations. Our respective techniques include incorporating the source code under test and the errors thrown by the latest generated test case to the LLM prompt. We assess our methods through a comparison study against the base UTGen version. We observe that source code retrieval enhances the generation of compilable test cases for complex classes. Error code retrieval shows similar hallucination performance to base UTGen, with a decrease in the number of re-prompts for classes with a high normalised Lack of Cohesion of Methods (*LCOM).

Index Terms - Automated Test Generation, Large Language Models (LLMs), LLM Hallucination, Prompt Engineering

Automated software testing plays a critical role in improving software quality and reducing manual testing expenses. However, generating understandable and meaningful unit tests remains challenging, especially with frameworks optimized for coverage like Search-Based Software Testing (SBST). Large Language Models (LLMs) have the capability to generate human-like text, while capture/replay techniques can provide realistic data scenarios through trace logs, contributing to meaningful test case generation. This study introduces UTGen+, an approach that enhances LLM-based SBST by integrating trace logs from end-to-end tests, aiming to further improve test case understandability.
We conducted a comparative user study with 9 participants using UTGen+, original UTGen, and conventional SBST (EvoSuite), focusing on the effects of trace log inclusion on the naturalness and relevancy of comments, identifiers, and test data across several projects. The results indicated that while UTGen+ did not improve the naturalness and relevancy of comments and identifiers, it significantly enhanced the relevancy of test data. These findings suggest that incorporating contextual data can indeed benefit the generation of more relevant and understandable automated test cases.

Since software testing is crucial, there has been research on generating test cases automatically. The problem is that the generated test cases can be hard to understand. Multiple factors play a role in understandability and one of them is test summarization, which provides an overview of the test of what it is exactly testing and sometimes highlights the key functionalities. There already exist some tools to generate test summaries that use template-based summarization techniques. Limitations of generated summaries include that they can be lengthy and redundant, and that it is best to use them in combination with well-defined test names and variables. There is a tool developed named UTGen, which combines Evosuite and Large Language models to increase understandability which includes improving the test names and variables, but does not have a summarization functionality yet. In this research, we extend UTGen using LLM-generated summaries. We investigate to what extent the understandability of a test case can be influenced by LLM-generated test summaries in terms of context, conciseness, and naturalness. For this reason, we do a user evaluation with 11 participants with a software testing background. They will judge LLM-generated summaries and compare them to existing summarization tools. The LLM-generated summaries scored overall higher than the template-based summaries and were also more favorable by the participants.

The advancement of Machine Learning (ML) in the last decade has created new business prospects for developers working on ML models. Models that are expensive and time-consuming to design and train can now be outsourced from others to reduce costs using Machine Learning as a service (MLaaS). \textbf{Deep Neural Networks (DNNs)} are particularly expensive to train, therefore many who need a DNN utilize the services of an MLaaS provider. This creates the \textbf{possibility of piracy} of this valuable asset, and the need to prevent piracy to assure a fair market. To address this need, research has been conducted on protecting DNNs using various watermarking techniques. A work by \textit{Li et al.} has proposed null-embedding, a technique that renders the DNN useless if it is subject to a piracy attack. Despite being effective, this method was shown to reduce classification performance when embedding a watermark into the model. This paper suggests modifications to the null-embedding technique that reduce this impact and keep the classification accuracy close to that of a non-watermarked model.

Data security has become more important over the last few years as data sharing over the world has become trivial. Data ownership therefore becomes critical as data can be very valuable and vulnerable to theft. Watermarking is a technique that can help data owners prove ownership over their data. In this paper, an approach is presented to watermark data that is gathered over time, such as weather data. With this research, we propose an adaptation of an existing audio watermarking technique developed in [1]. The adapted algorithm embeds a bit stream into a non-medical time series dataset by calculating the Discrete Wavelet Transform coefficients and modifying their magnitudes. The algorithm shows good robustness against a small range of data modification attacks but lacks capability in larger
scaled attacks. In addition, the proposed algorithm does require additional research to use it in a professional setting.

AI and machine learning have been topics of big interest in the last couple of years, with plenty of applications in many domains. To train these models into useful and desirable tools, a large amount of data is necessary. This data is expensive to collect, becoming one of the most valuable commodities of this century. As the value of data increases, protecting this intellectual property becomes more and more relevant. Watermarking is a technique widely used for data protection in media, but the non-media counterpart has not been researched as thoroughly. In this paper, an adaptation of a common watermarking technique, DWT watermarking, is applied on two datasets used for machine learning. This technique is invisible and robust in signal watermarking, but its performance on a numerical dataset has not been previously researched. A previously devised algorithm was used, but it was adjusted to better fit dataset watermarking. To assess the quality of the watermark, the marked data has been subjected to create, remove, update and zero-out attacks. On top of this, multiple machine-learning models have been trained on the marked data. Initial results show that the proposed technique performs well in terms of invisibility, obtaining similar or better accuracies than models trained on the original data, but it is quite sensitive to attacks. Even small modifications, less than 1\% of the data, can break the signature.

Large language models (LMs) are increasingly used in critical tasks, making it important that these models can be trusted. The confidence an LM assigns to its prediction is often used to indicate how much trust can be placed in that prediction. However, a high confidence can be incorrectly trusted if it turns out to be incorrect, also known as a high-confident error, or unknown unknown (UU). Blind spots, clusters of UUs, are caused by out-of-distribution data (OOD), bias, covariate shift, or unseen data. Previous work by Lippmann et al. generated samples based on these UUs, however, they selected a random subset of all found UUs. Although this was the first study that could mitigate blind spots without the need for crowd workers or oracles, they did not address the underlying causes of blind spots, such as OOD and bias. Data valuation is the research field that is concerned with valuing samples to improve model performance and can identify OOD data, bias, and noise. This paper proposed to combine the research field of data valuation, with the mitigation of blind spots. We generated synthetic samples using the highest-valued samples and retrained the LM using a Weighted Loss based on Data Values (WLDV). We conducted an extensive evaluation of our approach on four tasks, demonstrating a reduction of UUs by up to 32.7%, while retaining the same level of accuracy. This was the first exploration of combining the research fields of data valuation and blind spot mitigation.

In the realm of machine learning (ML), the need for efficiency in training processes is paramount. The conventional first step in an ML workflow involves collecting data from various sources and merging them into a single table, a process known as materialization, which can introduce inefficiencies caused by redundant data. Factorized ML strives to reduce this by maintaining the original data forms and performing model training on the separate source tables. This approach can lead to significant increases in training efficiency.

However, factorized training does not always reduce cost compared to traditional materialized training. This research tackles this issue by examining the multidimensional cost optimization problem that emerges when deciding between factorized and traditional materialized learning methods. It fills in gaps left by prior research, which is focused on CPU-based training, by investigating the cost estimation landscape for factorized ML, with a special emphasis on GPU performance compared to CPUs. The used factorized ML framework is expanded to incorporate GPU training, a topic not explored in previous research. We demonstrate that GPU training exhibits significantly different cost characteristics than CPU training, which has substantial implications for the design of cost models and the optimization of factorized ML.

Through an empirical study, an ML-based cost model is developed that can accurately predict the faster training method for a wide range of scenarios. On an extensive evaluation with real-world datasets this model boasts an average speedup of 3.8x, versus the state-of-the-art's 0.9x. We also show that it generalizes to scenarios with datasets and hardware settings on which the model is not trained, keeping 82% of training set performance.

Our innovative cost model for factorized ML enables significant time savings in training-intensive scenarios and further underlines the benefits of factorized training. However, effort should be invested into incorporating factorized training into existing ML frameworks so this method of training a model, and our cost model, can be evaluated in a larger set of realistic scenarios.

Online gaming is the world’s largest entertainment industry by revenue, and supports over 3 billion consumers worldwide. Many of the world’s most popular online games must manage millions of concurrent players through a single unified service. Achieving performant and scalable online games is challenging. Online games are subject to stringent quality of service requirements, notably extremely low response times, with at most 50ms being considered acceptable. Unlike many other types of applications, the performance of online games depends to a large degree on the resources available on end-user devices. These devices are typically heterogeneous, limited in compute and network resources, and subject to unpredictable changes in resource availability.
Addressing this challenge, we propose in this work the concept of differentiated deployment, which allows online games to selectively manage and scale online-game systems with fine granularity in response to changes in available resources. We design Polka, a framework for online games which supports differentiated deployment. We then implement PolkaDOTS, an open-source proof of concept of the Polka framework built in an industry standard game development ecosystem.
We evaluate our approach using Dither, a custom-built experiment runner for large scale distributed experiments on online games. We use Dither to perform real-world experiments on a representative Minecraft-like Game, Opencraft 2, built on the PolkaDOTS stack, and analyze the impact of various differentiated deployment scenarios. From these experiments, we find that differentiated deployment can decrease performance variability of online-game servers, and decrease the response time experienced by players by up to 32%. Most importantly, we show that differentiated deployment enables novel deployment techniques, including switching from local rendering to cloud-based rendering (i.e., cloud gaming) at runtime.

The current trend towards the integration of artificial intelligence (AI) and graphics processing unit (GPU) technologies has resulted in the development of embedded hybrid GPU-AI accelerators, which offer high computational power and energy efficiency. One of the key challenges in designing such accelerators is ensuring their timing correctness, as any timing violation may lead to system failure and incorrect results. To address this challenge, timing simulators have been proposed as a promising solution, as they enable accurate and efficient timing analysis of these complex systems. Nevertheless, such simulators have their limitations: detailed ones, such as cycle-accurate simulators, have high accuracy but exhibit high overhead, while faster approaches, like event-driven simulators, usually cannot achieve high accuracy levels. Therefore, the question arises: How can we implement a timing simulator to achieve a balanced trade-off between accuracy and execution time?

In this context, we introduce NEOX-V, a cutting-edge RISC-V based GPU Processor optimized for GPGPU and AI workloads. However, the current NEOX-V product lacks timing information in its simulator. This has prompted us to use it as a case study to bridge the gap between accuracy and execution time in timing simulators. This is achieved through the implementation of a new feature: a timing simulator that utilizes event-driven modeling.

To assess the proposed simulator's accuracy and effectiveness, we employ a comprehensive validation framework, using diverse workloads and configurations, from simple micro-benchmarks to intricate AI tests. The results demonstrate that the timing simulator achieves an accuracy error below 8\% when compared to the RTL equivalent for all applications, with a marginal increase in actual simulation time of only 0.7\%. It is worth noting that the timing simulator's utility extends beyond predicting execution time; it also plays a crucial role in verifying the existing design and uncovering its limitations.

Overall, this thesis makes a significant contribution to the field of computer architecture by providing a powerful tool for the design, development, and evaluation of an embedded hybrid GPU-AI accelerator called NEOX-V. It is our hope that this work will inspire further research and development in this exciting and rapidly evolving field.

SAGE is an unsupervised sequence learning pipeline that generates alert-driven attack graphs (AGs) without the need for prior expert knowledge about existing vulnerabilities and network topology. Using a suffix-based probabilistic deterministic finite automaton (S-PDFA), it accentuates infrequent high-severity alerts without discarding frequent low-severity alerts. It also captures the context of the alerts with identical signatures and it is an interpretable model. In order to deal with infrequent data, SAGE utilises sink states which are not merged during the S-PDFA learning process. However, this could result in unnecessarily larger AGs. In this study, we have looked at the AGs resulting from merging sink states with other sinks and the core of the S-PDFA after the main merging process. Data from Collegiate Penetration Testing Competitions has been used to compare AGs based on the four metrics: size, complexity, interpretability and completeness. We have shown that the resulting graphs are, on average, slightly smaller, with about the same complexity and the same completeness, but with worse interpretability due to losses of context of attack episodes, which cannot be compensated by the slightly smaller size of the AGs.

Intrusion Detection Systems (IDSes) detect malicious traffic in computer networks and generate a large volume of alerts, which cannot be processed manually. SAGE is a deterministic algorithm that works without a priori network/expert knowledge and can compress these alerts into attack graphs (AGs), modelling intruders’ paths in the network. These AGs are too high in quantity/complexity for manual analysis, creating the necessity for prioritising individual attack stages (ASes). The existing prioritisation metric does not take into account graph
properties and is not granular enough to function on a node level. We propose PICA, an urgency metric inspired by the CIA triad (Confidentiality, Integrity and Availability) and the graph properties. It works on a node level and an attack-stage level. PICA is evaluated by comparison with the current implementation, based on AGs generated by SAGE using open-source intrusion alert datasets. The evaluation is based on the number and the type of the discovered attack stages. Results show that PICA manages to discover ASes that contain nodes with a high
in-degree but fails at discovering urgent ASes that contain many nodes with low in-degrees. Compared to the baseline, the ASes are distributed more evenly over the different urgency levels. Analysis of urgent node positioning revealed that sub-AGs lose information when objectives (final goal in a path) are also starting nodes. Changing the weights of the CIA triad showed a clear bias in results
towards the larger weights, as was intended. Finally, further work is proposed for PICA and in the generation process of SAGE’s AGs.

The interpretability of an attack graph is a key principle as it reflects the difficulty of a specialist to take insights into attacker strategies. However, the quantification of interpretability is considered to be a subjective manner and complex attack graphs can be challenging to read and interpret. In this research paper, we propose a new metric for quantifying the interpretability of attack graphs, aiming for comparable results between attack graphs regardless of the chosen drawing configuration or generation method. We address the gap in existing metrics by combining elements from the theory of cognitive chunks of information and user-experience-related fields to measure interpretability in terms of cognitive load. Our metric leverages Gestalt principles to formalize the quantification of interpretability based on cognitive overload. Compared to a similar approach, the proposed metric reveals a high level of similarity with the baseline, however, qualitative analysis revealed the proposed metric eliminates certain discrepancies with the expert's opinion that the baseline metric presented. Furthermore, a use case of the metric is presented and we evaluate our metric by comparing attack graphs generated using different methods, such as deterministic finite automaton (S-PDFA), Markov chain, and suffix tree. Finally, further work is proposed toward the goal of completing the metric by incorporating the remaining Gestalt principles.

This research paper focuses on the complex domain of alert-driven attack graphs. SAGE is a tool which generates such attack graphs (AGs) by using a suffix-based probabilistic deterministic finite automaton (S-PDFA). One of the substantial properties of this algorithm is to detect infrequent severe alerts while maintaining the context of attacks via the help of sink states and state IDs. This is a modelling assumption that we validate by answering the question driving this research: How does allowing the sink states to merge with other sink states affect the generated alert-driven attack graphs? Our research used a transparent methodology to obtain size, complexity and completeness statistics of the two algorithms. Afterwards, outstanding values in size and complexity allow us to filter insightful attack graphs, subject to head-to-head comparisons concerning their interpretability. We discovered that many remarkable changes happen in the outputted attack graphs, leading to an evident decrease in interpretability and increased loss of context. Concurrently, we do not detect substantial changes in size, complexity and completeness, leading us to believe that it is possible to unlock SAGE's full potential by adding specific thresholds for merging sink states. One proposed constraint is allowing only the merges of sinks at equal distance to the victim node. This alteration leads to similar results in all metrics, including interpretability, where some AGs show improvements.

SAGE is a deterministic and unsupervised learning pipeline that can generate attack graphs from intrusion alerts without input knowledge from a security analyst. Using a suffix-based probabilistic deterministic finite automaton (S-PDFA), the system compresses over 1 million alerts into less than 500 attack graphs (AGs), which are concise and manageable. Unlike other frequency analysis methods, SAGE does not discard infrequent high-severity alerts, which are crucial for learning the penetration strategies of attackers. This paper compares the baseline algorithm (i.e. S-PDFA) with a modelling assumption generated by swapping the S-PDFA with a PDFA. The aim is to validate the quality of SAGE and propose possible solutions for PDFA usage, allowing the algorithm to generate AGs in real-time. We compare them both quantitatively and qualitatively using size, complexity, completeness and interpretability metrics. Our findings show that AGs generated by the PDFA are more readable and as complete while being slightly larger (i.e. 16% larger) than the baseline S-PDFA. In certain cases, it can also better capture different attack strategies, proving that, if further optimized, it can perform better than the baseline.

The introduction of cloud hosting has made it possible to elastically provision distributed stream processing systems (SPEs). By dynamically scaling the different operators of the system, resource consumption can be minimised while meeting the system service-level objectives. In the literature, many different auto-scaling techniques are proposed that make scaling decisions based on the current state of the system. However, these techniques are poorly evaluated and are rarely compared with each other. This makes it difficult to determine the state-of-the-art for auto-scaling techniques targeting SPEs, which slows down its development. In this paper, we design and implement a modular framework to evaluate the performance of state-of-the-art auto-scalers targeting SPEs. We implement state-of-the-art auto-scalers Dhalion, DS2, and Varga et al., using Kubernetes horizontal pod auto-scaler as baseline. We perform an end-to-end experimental evaluation of the auto-scalers and investigate their performance when run on different queries and workload patterns. Furthermore, we investigate the convergence time of the auto-scalers and evaluate their scaling accuracy. The results emphasise the difficulty of capturing the complex relationships of different operators and the struggle to balance resource efficiency and the performance of the system. Moreover, it shows the inherent weakness of reactive auto-scalers to react slowly to changing workloads and reveals the importance of considering the current health of the system when issuing scaling actions.