Late deliveries have been a common problem in the software industry for decades. They often result from deficiencies in effort estimation and project planning. These deficiencies arise due to the complexity of software development, where various social and technical factors affect project effort and scheduling. Variability in human elements, such as team dynamics and changing user requirements, adds further uncertainty. Since meeting time and cost estimates is crucial for project success, improving effort estimation and planning remains a key priority for software organizations. More accurate forecasting enables better resource allocation, reduces delays, and enhances customer satisfaction.

Over the past two decades, software organizations have increasingly adopted agile methods to improve flexibility and responsiveness. However, despite these advantages, schedule delays remain common, with nearly half of agile projects experiencing overruns of 25% or more. A key challenge lies in balancing the flexible, short-term planning of small functionalities (user stories) with the structured, long-term planning required for larger development units (epics). Current industry practices offer limited support for managing these complexities, especially in large-scale agile settings.

This thesis presents a novel suite of expert- and data-based strategies to improve effort estimation and planning in large-scale agile software development. We conduct a series of case studies at ING, a large Dutch internationally operating bank, to collect and analyze data from hundreds of agile teams and projects. We identify key factors influencing delays in epics and user stories and develop models to predict delays at both levels. At the epic level, we compile our findings into a conceptual framework representing influential factors and their relationships to on-time delivery. Additionally, we explore dynamic Bayesian methods to continuously update delay predictions throughout an epic's development life cycle. At the story level, we examine how team characteristics affect the likelihood of delays. We also investigate how these factors, combined with incremental learning methods, can improve story delay predictions. Finally, we develop a model that optimizes sprint plans based on team goals and delivery performance.

Our research identifies 25 factors and their interactions that affect the on-time delivery of epics. The most influential factors are predominantly social in nature, such as task dependencies, organizational alignment, and internal politics. These factors interact hierarchically: organizational factors shape team behavior, which in turn affects technical factors. To capture these complexities, we demonstrate that dynamic Bayesian methods, using delay patterns as input, effectively update delay predictions as new information becomes available. At the story level, our findings suggest that planning in agile settings can be significantly improved by integrating team-related information and incremental learning methods into predictive models. Moreover, we find that user story prioritization depends on a combination of factors that vary by project context. Our sprint plan optimization model effectively addresses this variability and generates plans that deliver more business value, align more closely with sprint goals, and mitigate delay risks better.

The increasing number of malicious packages being deployed in open source package repositories like PyPI or npm prompted numerous works aiming to secure open source ecosys- tems. The increased availability and deployment of safeguards raises the question whether and how attackers evolved their tactics and techniques, and the possibility of improving the current detection methods.
In order to improve detection, an important step is to understand how attackers design malicious packages and if it has evolved over time as detection methods improved. We conducted a semi-automated analysis and labeling of approx. 4000 malicious packages in order (a) to identify and quantify the techniques used by malware authors, (b) to uncover malware campaigns through clustering the CodeBERT embeddings of malicious code, and (c) to contrast the results with previous studies.
To detect malicious packages, existing work all rely on a common classification pipeline: first a human-defined rule-based feature extraction mechanism then a trained classifier. We hypothesize that the feature extraction mechanism is a limiting factor to classifier perfor- mance, and an additional step that needs to be kept up to date. To address this problem, multiple models utilizing the freshly labeled dataset were trained and compared: 2 types of fine tuned CodeBERT models, 2 isolation forest models and 1 out of distribution model.
We learned that the number of attacks on legitimate packages dropped in comparison to name confusion attacks, e.g. typo-squatting or dependency confusion, which are conducted with increasing campaign sizes. At the same time, packages are taken down more swiftly, whereas the characteristics of the malicious code itself did not substantially change. There are shifts in regards to the use of obfuscation or the primary objective, but the malicious code stays generally very simple.
We found that CodeBERT fine tuned with a loss function utilizing the equivalent of scikit-learn’s 𝑐𝑙𝑎𝑠𝑠_𝑤𝑒𝑖𝑔h𝑡 =′ 𝑏𝑎𝑙𝑎𝑛𝑐𝑒𝑑′ equation (Bert-Balanced) performed best with a specificity of 98.2% and recall of 93.7%. In the real world scenario, we evaluated a total of 8734 unique packages in the updates feed and 2181 in the new packages feed, and found 4 unique malicious packages. Bert-Balanced was able to keep up with the volume of packages being uploaded to PyPI, with plenty of computational resources still available.

Type annotations in Python are an integral part of static analysis. They can be used for code documentation, error detection and the development of cleaner architectures. By enhancing code quality, they contribute to the robustness, maintainability and comprehensibility of codebases. Tools like static type checkers use type annotations to detect bugs early, with some type checkers like Pyright being capable of inferring annotations statically from source code.

This thesis uses an innovative approach to further enhance type annotation coverage in Python codebases by using a combination of machine learning predictions and combinatorial search. To do this, PyHintSearch was developed. PyHintSearch constructs a search tree to which a depth-first search is applied to systematically explore potential combinations of predicted type annotations and validate them using feedback from the Pyright static type checker. Ultimately, the goal is to identify a branch containing a valid combination of type annotations. These annotations can then be integrated into Python code, thereby enhancing the type annotation coverage, which leads to improved static analysis and ultimately better code quality.

PyHintSearch's effectiveness is evaluated based on type annotation coverage and correctness, performance, and practical usability. Experimental results demonstrate different improvements in type annotation coverage, depending on the machine learning model used for type inference. Type4Py showed an improvement of 62.45% and TypeT5 of 79.93%. The precision of type annotations from these models are 0.36 and 0.51, respectively. Performance-wise, PyHintSearch can efficiently explore the exponential search space, annotating 16 diverse projects, ranging from small to large, in approximately 13.75 hours when using the Type4Py model. Regarding practical usability, the impact of type annotations on downstream program analysis is examined through the generation of call graphs. The additional information that type annotations provide can be used to refine the call graph by eliminating irrelevant calls to make it more precise.

Accurate topic classification is crucial in the scientific community when it comes to finding relevant journals. However, the efficiency and accuracy of topic classification of publications do not seem to be at its best performance, especially with the fast-paced rise in the quantity of research papers. Our research aims to address this problem by utilizing state-of-the-art (SOTA) methods. We chose the 'April 2022 Crossref' data set for the research, as Alexandria3k, the tool utilized for querying on the open data set, is tested on the same data. We stratified 50,000 data that have title, abstract, and work names, which are the roughly assigned topics. SOTA methods chosen for feature extraction and classification models are OpenAI Embeddings and XGBoost. Our research shows that this combination of SOTA methods has the potential to improve the performance of current topic classification of publications.

The growth of academic publications, heterogeneity of datasets and the absence of a globally accepted organization identifier introduce the challenge of affiliation disambiguation in bibliographic databases. In this paper, we create a baseline using the currently implemented algorithm for author affiliation linkage in Alexandria3k by comparing it to the ground truth. We aim to explore the usage of LLMs (GPT-4) in the Alexandria3k environment to disambiguate author affiliations. The proposed approach extracts the research organization from textual affiliations provided by researchers through their published works and cross-references the organization across the Research Organization Registry. Our process shows promising results and a significant improvement on the existing algorithm in terms of matching rate and identification of multiple affiliations. We discuss the margin of error in LLM results, limitations of the ground truth, and suggest future research directions.

The Alexandria3K library, a versatile Python-based tool, has been expanded to include the integra- tion of the PubMed dataset, enriching its capabil- ities in the analysis of scientific papers. Origi- nally supporting major datasets like Crossref and US patents, and smaller yet significant datasets like ORCID. The addition of PubMed enables in-depth analysis of medical papers, with medical specific data and articles not yet in the Crossref dataset. This research focused on validating the integration of PubMed into Alexandria3K. To achieve this, two literature surveys were replicated using the com- plete PubMed dataset. The first survey involved querying different pathogens in the dataset for three regions. The results were comparable, although some articles were missed by Alexandria3K but two articles were also missed by the original sur- vey. The second survey revolved around software tools used in medical papers. Although fewer ar- ticles were found with Alexandria3K the ratio for most tools was still comparable. Although a thor- ough manual review of all articles could have fur- ther refined the reevaluation, time constraints pre- vented this step. These replicated surveys demon- strate Alexandria3K’s potential in conducting lit- erature surveys, underscoring the need for manual validation to complement its capabilities.

Author name disambiguation, otherwise described as (publication) record linking, is a problem that has had considerable research dedicated to its solv- ing. Author attributions, calculating research met- rics and conducting literature reviews are amongst processes that experience increased difficulty due to ambiguous author names. In this study, a novel approach is presented to disambiguate au- thors related to scientific publications, using Large Language Models (LLMs) in combination with the Alexandria3k software package. LLMs have shown great potential in processing, analysing and drawing conclusions when presented with human- readable data. The approach presented in this study supplies a LLM with known attributes of publica- tion records and authors, such as names, affiliations and co-authors, to determine whether records writ- ten by authors with ambiguous names can be linked to the same real-world person. Using Alexan- dria3k, a dataset of authors and publications with confirmed identities is created to test and validate the approach. Finally, the approach is measured against state-of-the-art methods to disambiguate author names and different configurations are pre- sented and discussed.

Type inference plays a pivotal role in modern software development as it aids in understanding code, detecting errors, and facilitating code completion. Two main approaches, static analysis, and machine learning, contribute to this process. Each approach has its own benefits and limitations. This thesis investigates the potential of combining static analysis techniques and machine learning (ML) approaches to enhance type inference capabilities.

The adoption of the serverless architecture and the Function-as-a-Service model has significantly increased in recent years, with more enterprises migrating their software and hardware to the cloud. However, most applications require state management, leading to the use of external databases. To alleviate the burden of state management, there are systems known as SFaaS (Stateful Function-as-a-Service) that provide stateful functions. Despite their benefits, SFaaS systems still face challenges such as the need for transactional logic. Stateful streaming dataflow engines offer promising capabilities for implementing transactional SFaaS systems due to their exactly-once message delivery guarantees and global state management. Key-value stores serve as embedded databases in this architecture, making it crucial to carefully evaluate available options for suitable types of key-value stores.
This work focuses on the implementation and evaluation of three distinct types of log-structured key-value stores within the context of serving as state-management backends for transactional dataflow systems. A key aspect of our implementations is the incorporation of efficient incremental snapshotting functionality. We explore the performance and suitability of these key-value stores in managing state and supporting transactional operations in dataflow systems.

The use of open-source packages is a common practice among developers. It decreases the development time and improves maintainability. But adding a dependency to a project comes with inherit risks such as introducing vulnerabilities. A few solutions that help visualize all of the dependencies of a project exist already. However, none provide the capability of selecting a moment in time for analyzing the generated structure. This research paper formalizes a time-based dependency graph that can be generalized to any ecosystem and then showcases its usefulness by analyzing the Python ecosystem throughout time. The results indicate that the Python ecosystem does have a subset of packages that are the most used - such as numpy and requests - but overall it is well balanced, meaning that it is able to withstand the removal of one of its most used packages. The data structure also provides satisfactory results, having a 89.6% accuracy when compared to the Python resolver. The findings of this study can be used to improve existing dependency networks.

In (open-source) development, developers routinely rely on other libraries to improve their coding efficiency by reusing code. This reliance on other packages could cause issues when critical dependencies have suddenly have a vulnerability introduced to them. This work analyzes the criticality for NPM. To get an accurate picture of what the most-critical and thus possibly most-vulnerable packages are, the entirety of NPM must be analyzed. However, this proved too big to be able to fit in 500GB of memory. This work therefore examines a small subset of 100 thousand packages. To do the analysis, this paper proposes a novel approach of embedding a time dimension into the package network to provide better accuracy. This papers analysis show that both with and without this time dimension, \texttt{babel} packages are by far the most important in the package graph (as measured by PageRank). We should, however, keep in mind that this came from only analyzing 100 thousand packages. Thus, further research is required to confirm this conclusion. In particular, other importance measures should be used to find out the packages' criticality.

Using open-source packages when developing software applications is the general practice among a vast amount of software developers. However, importing open-source code which may depend on other existing technologies may lead to the appearance of a transitive dependency chain. As a result, failure of packages with a high amount of transitive dependants may have an impact on the performance of all dependant applications. This work focuses on designing a graph data structure which maps the dependency relationship between packages as edges, with nodes representing a single version of a certain package. Moreover, the data structure may perform queries based on time intervals, being able to resolve versions in the same manner as a package distributor would. By constructing such data structure, an analysis of the most critical packages for an ecosystem can be conducted. This paper looks mainly into the Debian ecosystem, and searches for applications which are most critical. Based on this paper criticality evaluation, the package dpkg was found to be both most critical and most used in whole Debian's package main repository.

The main principle of Open Source development is that developers can reuse different libraries over and over again to make their lives easier. That is why this practice has gained a lot of popularity. However, libraries usually depend on other already existing pieces of code. This means that whenever some small piece of code fails, the whole application may crash due to its dependencies. Since Debian is considered as one of the largest community run distributions, it is important to have a good tool to analyse these dependencies to help in avoiding such crashes. That is why this research will focus on building a dependency graph for Debian's package manager and finding out which are the most widely used packages. What separates this research from previous related works, is the addition of a time component to the graph. This will be in the form of a timestamp of the release date for each version of a package. This allows for extensive traversal of the graph, which can also be based on time periods. By doing so, no transitive dependencies should be missed. The paper concludes with defining most widely used as the packages which achieve the highest PageRank score when put into the graph. The top 3 ones are "libc6", "libgcc1" and "multiarch-support".

Developers rely on different software to improve their efficiency as to reuse parts of code and be able to maintain it with ease, which is why open source software libraries have gained much pop- ularity over the past years. This paper analyzes what are the most used packages from Apache Maven, which is a build automation tool used primarily for Java projects. In order to do an accurate analysis, it is necessary to collect all the packages and their dependencies from the Maven Central Repository. But, as of 5th of May 2022, analyzing the whole repository proves to be an extreme challenge, as the repository has 9M of indexed artifacts, which contain metadata such as the version and a list of dependencies. Besides, there are many more other repositories that also contain such artifacts. Thus, this pa- per examines only a small subset of artifacts and provides a data structure that is able to take into consideration the time dependency between li- braries, which can be also scaled to suit a much bigger input. The paper concludes with the most used software on given time-frames, but with- out specifying any time frames, the popular junit package comes out first.

The appeal of delivering new features faster has led many software projects to change their development processes towards rapid release models. Even though rapid releases are increasingly being adopted in open-source and commercial software, it is not well understood what the effects are of this practice. This thesis presents an exploratory case study of rapid releases at ING, a large banking company that develops software solutions in-house, to characterize rapid releases. Since 2011, ING has shifted to a rapid release model. This switch has resulted in a mixed environment of 611 teams releasing relatively fast and slow. We followed a mixed-methods approach in which we conducted a survey with 461 participants and corroborated their perceptions with two years of code quality data and one year of release delay data. Our research shows that: rapid releases can be beneficial in terms of code reviewing and user-perceived quality; rapidly released software tends to have a higher code churn, higher test coverage and lower average complexity; rapid releases are perceived to be, and are in fact, more commonly delayed than their non-rapid counterparts; however, rapid releases are correlated with shorter delays (median: 6 days) than non-rapid releases (median: 16 days); challenges in rapid releases are related to managing dependencies and certain code aspects, e.g. design debt. Based on our findings we present challenging areas that require further attention, both in practice and in research, in order to move the practice of rapid releases forward.

The number of conducted A/B tests is growing throughout companies in software development. Many of these companies develop their own in-house Experimentation Platform to support these experiments. In this thesis we identify factors that influence the trustworthiness and soundness of A/B tests by conducting a literature review. We discuss nineteen influential factors categorised as essentials and pitfalls. Using the data of 268 experiments from ING we verify the trustworthiness of ING’s own Experiment Platform, and conclude that there is room for improvement. Finally, we provide a method for developers and engineers to consider these factors during the experimentation phase by modelling them into a questionnaire containing 67 questions. These questions are grouped into three categories, which are referred to as the three A’s of A/B Test validation: Availability, Analysability and Accuracy. To help administer this questionnaire we introduce the first Open-Source toolkit for this matter: ABvalidator.

Training machine learning (ML) models for natural language processing usually requires lots of data that is often acquired through crowdsourcing. In crowdsourcing, crowd workers annotate data samples according to one or more properties, such as the sentiment of a sentence, the violence of a video segment, the aesthetics of an image, ... To ensure quality of the annotations, several workers annotate the same sample, and their annotations are combined into one unique label using aggregation techniques such as majority voting.

When the property to be annotated by the workers is subjective, the workers’ annotations for one same sample might differ, but all be valid. The way the annotations are aggregated can have an effect on the fairness of the outputs of the trained model. For example only accounting for the majority vote leads to ignoring the workers’ opinions which differ from the majority and consequently being discriminative towards certain workers. Also, ML models are not always designed to account for individual opinions, for simplicity's or performance's sake. Finally, to the best of our knowledge, no method exists to assess the fairness of a ML algorithm predicting a subjective property. In this thesis we address such limitations by seeking an answer to the following research question: how can targeted crowdsourcing be used to increase the fairness of ML algorithms trained for subjective properties' prediction?

We investigate how annotation aggregation via majority voting creates a dataset bias towards the majority opinion, and how this dataset bias in combination with the current limits of ML models lead to an algorithmic bias of the ML models trained with this dataset and unfairness in the model’s outputs. We assume that an ML model able to return each annotation of each user is a fair model. We propose a new evaluation method of the ML models' fairness, and a methodology to highlight and mitigate potential unfairness based on the creation of adapted training datasets and ML models.
Although our work is applicable to any kind of label aggregation for any data subject to multiple interpretations, we focus on the effects of the bias introduced by majority voting for the task of predicting sentence toxicity.

Our results show that the fairness evaluation method that we create enables to identify unfair algorithms and compare algorithmic fairness, and the final fairness metric is usable in the training process of ML models. The experiments on the models point out that we can mitigate the biases resulting from majority voting and increase the fairness towards the minority opinions. This is provided that the workers’ individual information and each of their annotations are taken into account when training adapted models, rather than only relying on the aggregated annotations, and that the dataset is resampled on criteria according to the favoured aspect of fairness. We also highlight that more work needs to be done to develop crowdsourcing methods to collect high-quality annotations of subjective properties, possibly at low-cost.

With the increase of online education, a good description of learning resources has become vital for educational resource sharing and reuse. Resource description has been under the spotlight in recent years. Educational platforms can benefit from good resource organisation and description, thereby providing a higher quality of services and attracting more learners to use their systems. Furthermore, well-described resources with metadata, promote content sharing and re-use.
This work starts with an extensive literature research on metadata generation techniques and breaks the findings down to metadata types. A detailed taxonomy of metadata types, based on this research, is provided. The taxonomy takes into account properties common to these types. Second, this work analyzes the state-of-the-art metadata collection techniques in literature and real-world educational content repositories including a showcase with the TUDelft library, in order to estimate the gap of metadata employment in the field of education. Following the results of this research and based on the observation that similar steps are often performed together, a set of easy-to-follow and generic enough design patters for generating metadata was identified. These design patterns aim at assisting content authors or data professionals with filling in metadata and thereafter, allowing for feature development or improvement in the respective platforms. The patterns for metadata extraction are based on the identified taxonomy of metadata. Finally, semantic metadata is extracted as proof of concept for two of the proposed patterns. A satisfactory to a high-quality result was achieved, showing that the patterns are intuitive and the data extracted with them, can be potentially used to describe the respective Educational Resource (ER) by adding the extracted information to its metadata.

In this project we aimed to create a post-trading day safeguard system that allows for the identification of bugs in the primary and secondary risk control systems at Optiver. These systems are needed to prevent undesirable exposure to the market from happening, and to ensure that they know exactly what this exposure is. The amount of input data for this project, given in the form of log files, equates to roughly 200 GB per trading day, post sanitation. We have developed a program that can simulate an entire trading day and detect if any limits were breached. This program can be run overnight, allowing for a T+1 report in the morning after the respective trading day. The difficulties in this project were in the acquisition of all knowledge concerning the unique traits of various markets around the world, inconsistencies in the data, incomplete documentation, and optimization of the program to run within the required time.

Organizationally, the project was executed within an agile workflow, with Kanban as software development methodology. Furthermore, the project is tested extensively to ensure the accuracy and correctness of the program.

Concerning the impact of the project, it contributed to the identification and resolution of multiple previously unknown bugs in the control systems at Optiver. Furthermore, our project verified the existence of some previously known issues. In the future, when the software is run to verify all order logs of Optiver, the software will prove its value by either increasing the confidence that there is an absence of bugs in the RiskGuard and autotrading software of Optiver or by identifying breached limits, indicating a bug.

Reactive Programming is a way of programming designed to provide developers with the right abstractions for creating systems that use streams of data. Traditional debug tools lack support for the abstractions provided, causing developers to fallback to the most rudimentary debug tool available: printf-debugging. In this work, we design a visualization and debugging tool for Reactive Programming, that aids comprehension and debugging of reactive systems, by visualizing the dependencies and structure of the data flow, and the data inside the flow. We present RxFiddle, a platform for the visualization as well as the required instrumentation for RxJS in the ReactiveX-family of Reactive Programming libraries. Evaluation based on an experiment with 111 subjects, shows that RxFiddle can outperform traditional debugging in terms of debug time required.