We explored the effect of augmenting a standard language model’s architecture (BERT) with a structural component based on the Abstract Syntax Trees (ASTs) of the source code. We created a universal abstract syntax tree structure that can be applied to multiple languages to enable the model to work in a multilingual setting. We adapted the general graph transformer architecture to function as the structural component of the transformer. Furthermore, we extended the Embeddings from Language Models (ELMo) style embeddings to work in a multilingual setting when working with incomplete source code. The final results showed that the multilingual setting was beneficial to achieving higher quality embeddings for the embedding model, however, monolingual models performed better in most cases for the transformer model. The addition of ASTs resulted in increased performance in the best performing models on all languages, while also reducing the need for a pre-training task to achieve the best performance. The largest increase in performance for a Java model compared to its baseline counterpart was 3.0% on average on the test set, the largest increase in performance for a Julia model compared to its baseline counterpart was 1.1% on average on the test set, and the largest increase in performance of a CPP model compared to its baseline counterpart was 5.7% on average on the test set. ... Read more

As software systems evolve over time, the quality of its structure and code degrade unless developers regularly maintain it, requiring significant effort. Automated tools to help developers maintain software have been well-studied in the past.
In particular, software remodularisation tools focus on improving the code structure quality with minimal effort by suggesting changes to the developers to obtain an improved modularisation.
While there has been considerable research on automated software remodularisation, it often faces one or more of the following three shortcomings.
First, the approach is applied to small or medium-size codebases, raising the question of whether it scales to large codebases. Second, the results are not validated by the developers of these codebases. Last, the algorithm optimises only from a code quality metrics point of view, not considering the perspective and knowledge of developers. In this thesis, we propose an approach to capture developers' domain knowledge of a large-scale object-oriented codebase, which uses an NSGA-III algorithm to suggest remodularisations that improve code structure quality and adhere to developer knowledge. Additionally, the results of the algorithm are validated by the developers. The results in this thesis show that with little effort, the domain knowledge of developers can be captured and used to improve the suggestions made by the algorithm. ... Read more

Finding defects in proposed changes is one of the biggest motivations and expected outcomes of code review, but does not result as often as expected in actually finding defects. Just-in-time (JIT) defect prediction focuses on predicting bug-introducing changes, which can help with efficient allocation of inspection time according to the defect-proneness of the changed software parts. Despite the promising results achieved by DeepJIT and CC2Vec, two deep learning-based JIT defect prediction models, industry-based JIT defect prediction studies have not opted yet to apply deep models. In this work, the goal is to build and evaluate several JIT defect prediction models that can help Adyen developers spot defective changes during code review. To construct a new dataset with a large enough set of labels, we identify four sources of potential bug-fixing commits by analysing Adyen's way of working. We make several practical adaptations to DeepJIT and CC2Vec and compare their performances with three traditional metric-based models when making predictions at both commit-level and file-level. Our results indicate that deep models are able to outperform the metric-based models across all three datasets. All models performed slightly worse when evaluated on Adyen data compared to an open-source setting, but both deep models still achieved respectable performances and significantly outperformed the metric-based models. When evaluated in a real-world setting on bugs manually collected by Adyen developers, DeepJIT performed consistent with earlier findings when evaluated on commit-level, but performances fall on file-level. Lastly, we find that although inclusion of each bug source generally does not lead to worse performance, whether it leads to better performance is dependent on both what type of model is used and at what granularity predictions are made. ... Read more

While SQL engines are now capable of detecting a large number of syntactic mistakes, most often semantic errors are not detected, which can lead to serious performance issues or even security vulnerabilities being introduced in the system. This thesis proposes a set of 25 validated heuristics together with a new rule-based static analysis tool for detecting the most common types of semantic bugs in SQL queries, based on evidence from previous research. We conduct an empirical study on the prevalence of semantic bugs in SQL on two datasets with queries collected from different open-source industry projects as well as on a large dataset of queries collected from StackOverflow posts. Manual analysis of more than 500 queries shows that our tool is able to detect semantic bugs in SQL queries with an accuracy of 97%. Furthermore, out of all 191,994 collected queries, we identified a total of 36,818 queries which contain at least one semantic bug, meaning that 19.17% of queries contained some semantic problem in their formulation. To the best of our knowledge, this is the largest dataset of SQL queries extracted from StackOverflow and could later be used for subsequent studies as well. ... Read more

Logging is a common practice in software development that assists developers with the maintenance of software. Logging a system optimally is a challenging task, thus Li et al. have proposed a state-of-the-art log recommendation model. However, no further attempts exist to improve the model or reproduce their results using different training data. In this research, a model was developed using the methods of Li et al. to evaluate its performance when trained on a specific dataset. Some aspects of the model such as feature filtering were studied. It was concluded that the methods of Li et al. are reproducible and can produce a model that performs well with various training data. The study on feature filtering revealed that not filtering features results in an increase of all tested metrics. ... Read more

Logging is a common practice of vital importance that enables developers to collect runtime information from a system. This information is then used to monitor a system's performance as it runs in production and to detect the cause of system failures. Besides its importance, logging is still a manual and difficult process. Developers rely on their experience and domain expertise in order to decide where to put log statements. In this paper, we tried to automatically suggest log placement by treating code as plain text that is derived from a vocabulary. Intuitively, we believe that the Code Vocabulary can indicate whether a code snippet should be logged or not. In order to validate this hypothesis, we trained machine learning models based solely on the Code Vocabulary in order to suggest log placement at method level. We also studied which words of the Code Vocabulary are more important when it comes to deciding where to put log statements. We evaluated our experiments on three open source systems and we found that i) The Code Vocabulary is a great source of training data when it comes to suggesting log placement at method level, ii) Classifiers trained solely on Vocabulary data are hard to interpret as there are no words in the Code Vocabulary significantly more valuable than others.
... Read more

Automated log level recommendation is a growing area of research in the field of logging. Logs are essential in software maintenance. Log levels influence the severity of the logs being printed. Recent studies have investigated different metrics for automated log level recommendation. Recently, a paper was published using automated deep learning based on syntactic context features for log level recommendation. The paper shows promising results, both for within-system evaluations and cross-system evaluations. Here, the results posed by that paper are validated by reconstructing the model from the paper. Furthermore, the model performance is evaluated on different features, for
instance, the containing block type. This study demonstrates that automated deep learning based on syntactic context features for log level recommendation certainly provides promising results. The outcomes even indicate that cross-system performance resembles within-system performance. However, this paper also indicates that the model cannot predict log levels for unseen systems. In conclusion, this paper validates that the current methodologies show potential for future research, but that the model is not ready for production. More research is necessary to transform the current algorithm into a production ready version of the algorithm. ... Read more

Since their invention, the keyboard and mouse are the most used input devices that software developers use to interact with source code. However, these devices have their IDE interaction issues as developers need to spend a significant amount of time learning how to use them efficiently and effectively. To tackle some of these issues, we used an eye-tracker to provide an alternative input method. With eye-tracking, it is possible to infer where developers are putting their attention to that can be utilized to improve the IDE user experience and productivity. Therefore, we present EyeIDEA, an experimental plugin for Intellij IDEA that integrates eye-tracking to provide source code navigation, debugging interactions, and fine-grained information about user habits based on eye-tracking. A user study was conducted with TU Delft students to investigate the perceived usefulness of our eye-tracking IDE. Based on their reactions, we found that the gaze-based interactions feel quick and natural. Moreover, there is a strong preference for eye-tracking interactions that only require a single code element or button as input. Additionally, users perceived that an eye-tracking IDE could substitute a
mouse, especially when suffering from health-related issues or when performing light programming work. However, eye-tracking also brings new challenges, including integrating eye-tracking with keyboard/mouse input and accounting for accuracy and precision issues that influence the overall usability. A short video demonstration of our tool is available at https://www.youtube.com/watch?v=ShvIX04rcr4. ... Read more

Refactoring is the process of improving the structure of code without changing its functionality. The process is beneficial for software quality but challenges remain for identifying refactoring opportunities. This work employs machine learning to predict the application of the refactoring type Extract Method in an industry setting with the use of code quality metrics. We detect 919 examples in industry code of Extract Method and 986 examples where Extract Method was not applied and compare this to open-source code. We find that feature distributions between industry and open-source code differ, especially in class-level metrics. We train models to predict Extract Method in industry code and find that Random Forests perform best. We find that class-level metrics are most important for the performance of these models. We then investigate whether models trained on an open-source set generalize to an industry setting. We find that, although less performant than a custom fit model, a Logistic Regression type model performs admirably. Afterward, we examine whether these models perform on unseen industry projects by validating on projects excluded from the training set. We find that average performance is decent but lower than when using the whole industry dataset or an open-source dataset for training. Lastly, we conduct a blind user study in which we ask experts to judge predictions made by our best model. We find that experts generally agree with the model's predictions. In the case that experts agree with the model's prediction to apply Extract Method, they do so because of high code complexity. When they agree with the model's prediction not to refactor they most frequently give the reason that the respective methods are already sufficiently understandable. ... Read more

Many development models exist, but finding which one is the right for a specific project or software company is difficult. Every project has its requirements and might need its own development model. The most popular development models are trunk-based development and merge requests.
There are no clear science-based guidelines on when to adopt one model or the other and challenges that teams face when migrating from one to another.
We perform a field study as this master thesis aims to provide more understanding on the impact of migrating from one development model to another at a large company. More specifically, a migration from trunk-based development to merge request-based development at a large software engineering company.
During this research, we interview 19 developers, eleven before the migration and eight after the migration, survey 46 developers to triangulate our findings of the interview before the migration, and analyzed the differences in the code reviews made by developers before and after the migration.
We show what benefits and challenges developers experience using the trunk-based model and what they expect from the merge request-based model before the migration. Also, we show the change of motivation for code reviews after the migration. Moreover, quantitative data shows that code reviews are completed faster and with more code comments in the merge request-based model. Finally, we provide the perceptions of developers after the migration. ... Read more

Mistakes in binary conditions are a source of error in many software systems. They happen when developers use < or > instead of <= or >=. These boundary mistakes are hard to find for developers and pose a manual labor-intensive work. While researches have been proposing solutions to identify errors in boundary conditions, the problem remains a challenge. In this thesis, we propose deep learning models to learn mistakes in boundary conditions and train our model on approximately 1.6M examples with faults in different boundary conditions. We achieve an accuracy of 85.06%, a precision of 85.23% and a recall of 84.82% on a controlled dataset. Additionally, we perform tests on 41 real-world boundary condition bugs found from GitHub and try to find bugs from the Java project of Adyen. However, the false-positive rate of the model remains an issue. We hope that this work paves the way for future developments in using deep learning models for defect prediction. ... Read more

Web APIs can have constraints on parameters, such that not all parameters are either always required or always optional. Sometimes the presence or value of one parameter could cause another parameter to be required. Additionally, parameters could have restrictions on what kinds of values are valid. We refer to these as inter-parameter and single-parameter constraints respectively. Having a clear overview of the constraints can help API consumers to integrate without the need for additional support and with fewer integration faults.

We developed two approaches for identifying parameter constraints in complex web APIs. One approach uses online documentation to infer inter-parameter constraints, the other depends on static code analysis to extract inter- and single-parameter constraints from the control flow of the API’s source code. In our case study at several APIs at Adyen, the documentation- and code-based approach can identify 21% and 53% percent of the constraints respectively. When the constraints identified by both approaches are combined, 66% of the inter-parameter constraints can be identified. Code analysis is able to identify 78% of the single-parameter constraints. ... Read more

Code quality of software products often degrades while they grow. Counteracting the degradation of code quality or improving it requires immense effort. Tools that reduce this effort are a hot topic in software engineering research. Software Modularization in particular aims to aid in the process of improving the quality of code structure, by finding flaws in code structure and suggesting improvements. Much research has been done in this field, however, most of it is applied on small to medium scale codebases. In addition, the quality of solutions implied by this research is often not properly validated. This thesis aims to apply an existing approach to an enterprise-level codebase, namely that of Adyen, and validating the results with developers experienced with the code. We achieve this by taking a graph-based approach, applying the NSGA-II algorithm, and introducing a new metric called the Estimated Build Cost of module Cache Breaks. We evaluate the approach in two ways. First, we performed a controlled experiment exploring the feasibility of the approach on larger scale codebases. For this experiment, we apply the approach to the Adyen codebase. The results show that the approach is scalable and shows a significant improvement of code quality in terms of the metrics used. We then performed a user study where we explore the feasibility of these results in practice. For this study, solutions generated during the experiment are split up and filtered to form groups with less than 10 changes, which are reviewed by developers that have changed that code recently or are a senior with experience in that specific area of the code. The results show that the algorithm is successful in identifying flaws in the codebase. However, the improvements it suggests are less precise and require future work. ... Read more

The internet traffic is constantly rising nowadays due to the significant increase of the devices connected to the Internet. As a consequence, many cyber risks have arisen. Cybercriminals are trying to exploit the vulnerabilities of these devices to cause damage and gain profit. Monitoring the network traffic and detecting such threats has become essential in order to keep safe systems that are connected to the Internet. The powerful properties of state machines and the sequential nature of the network traffic data, makes them an interesting and promising solution for the implementation of an intrusion detection system.

The goal of this thesis is to implement a new state-merging heuristic which will speedup the state machine building procedure without a significant loss on the quality of the model, and use it to detect malicious host on network traffic data. The new state-merging heuristic is utilizing the Locality-sensitive Hashing concept to store the future traces of each state and simplify the consistency check for the merge of two states. The network traffic data used are in the NetFlow format, and they are encoded and converted into traces in order to build the state machine model and measure its performance. The state machine built is modeling a malicious behavior and used to classify other hosts.

We show that the models built can effectively detect the malicious hosts, with its performance being comparable to the one of a state-of-the-art model. At the same time, the time needed to build the model is much less when compared to the time needed by other state-merging heuristics. ... Read more

It has become a common practice for software developers to analyse monitoring data as a means to understand issues, trends and performance of large-scale software systems. The combination of this practice with the software development workflow can however be cumbersome at times. One recent effort into evaluating the effects of integrating monitoring directly into the software development workflow has been done at the large-scale payment company Adyen in cooperation with the TU Delft. This project is a new attempt at making a flexible, open-source and useful monitoring tool based on the results of that research effort. This report will detail how the system is designed, implemented and the development methodology that was used for its creation. After researching popular monitoring stacks and solutions, an architecture for the Hyperion pipeline was designed that would maximize compatibility with logging setups. The monitoring data is first retrieved from a log source like Logstash or Elasticsearch. It is preprocessed by a pipeline consisting of configurable and flexible plugins. A large range of plugins is supported: functionalities include debugging, transformation, load balancing, and adding new data from third-party systems. There is also extensive support for third-party developers to create their own pipeline plugin to manipulate data in any way required, which allows Hyperion to account for unconventional logging setups. The pipeline typically ends at some type of aggregator, which exposes an interface for developer tools to retrieve metrics. These metrics can be queried with the Hyperion IDE plugin, which visualizes them in the development environment itself. To verify that the designed Hyperion pipeline adheres to the requirements defined in the research report, validation tests have been performed on every module of the system. All code is extensively tested with unit, integration, and container-based system tests. An evaluation of the performance of the pipeline additionally verified the scalability of the system even when provided with a large number of incoming logs. Finally, in order to ensure easy extension and adoption of the Hyperion pipeline a large amount of documentation was written, that explains the architecture of the project and how a third-party developer can adjust and extend it. ... Read more

Although writing code seems trivial at times, problems arise when humans misinterpret what source code actually does. One of the potential causes are “atoms of confusion”; the smallest possible patterns of misinterpretable source code. The misunderstandings and errors have been studied in past for the C programming language. They are found to occur in many large projects and style guides. In this work, the existing tested set of atoms of confusion has been translated to Java. With this new set, our aim was to find out what atoms of confusion hinder the comprehensibility of Java programs. Additionally, we wanted to find out how these confusion patterns are perceived. To this end, the new code snippets are used in a two­fold experiment. The first part of the experiment asked the participant to write down the output of a code snippet. The results of this showed us that 7 out of the 14 translatable atoms are the cause of misunderstandings for students. We measured a significant increase in mistakes caused by the atoms of confusion. In the second part of the experiment we asked the participants to compare two code snippets on how confusing they are. One code snippet included the confusing pattern, while in the other, the pattern is avoided. Results showed us that these students also perceive the atoms of confusion as being more difficult to understand. The combined results show us the significance of these atoms of confusion, and show us examples of situations where we cannot assume programmers to simply grasp the meaning of what we write down as code. The code snippets for the experiment, scripts, and data used for this experiment are provided in an online appendix. ... Read more

Software maintenance is an essential and time-consuming task during the software development cycle. Readability of test code is a crucial element for performing programming tasks, such as testing, bug fixing and maintaining code. Hence poorly written tests are difficult to maintain and lose their value to developers. In order to overcome this problem, we need to understand how programmers read the test code. Therefore we conducted an empirical study to analyze the various reading patterns in novices and professionals using a sophisticated eye tracking device. Our results show that (i) all programmers first comprehended the production code and then switched between test and production codes, (ii) novices had higher fixations reading test code and assert statements, (iii) professionals revisited the test code more than novices, (iv) professionals had significantly lesser test code coverage than novices, and (v) there is a significant difference in reading test code between novice and professionals. ... Read more

Refactorings tackle the challenge of architectural degradation of object-oriented software projects by improving its internal structure without changing the behavior. Refactorings improve software quality and maintainability if applied correctly. However, identifying refactoring opportunities is a challenging problem for developers and researchers alike. In a recent work, machine learning algorithms have shown great potential to solve this problem. This thesis used RefactoringMiner to detect refactorings in open-source Java projects and computed code metrics by static analysis. We defined the refactoring opportunity detection problem as a binary classification problem and deployed machine learning algorithms to solve it. The models classify between a specific refactoring type and a stable class using the metrics as features. Multiple machine learning experiments were designed based on the results of an empirical study of the refactorings. For this work, we created the largest data set of refactorings in Java source code to date, including 92800 open-source projects from GitHub with a total of 33.67 million refactoring samples. The data analysis revealed that Class- and Package-Level refactorings occur most frequently in early development stages of a class, Method- and Variable-Level refactorings are applied uniformly during the development of a class. The machine learning models achieve high performance ranging from 80\% to 89\% total average accuracy for different configurations of the refactoring opportunity prediction problem on unseen projects. Selecting a high Stable Commit Threshold (K) improves the recall of the models significantly, but also strongly reduces the generalizability of the models. The Random Forest (RF) classifier shows great potential for the refactoring opportunity detection, it can adapt to various configurations of the problem, identifies a large variety of relevant metrics in the data and is able to distinguish different refactoring types. This work shows that for solving the refactoring opportunity detection problem a large variety of metrics is required, as a small set of metrics cannot represent the complexity of the problem. ... Read more

Monitoring software behaviour is being done in various ways. Log messages are being output by almost any kind of running software system. Therefore, learning how software behaves from doing analysis over log data can lead to new insights about the system. However, the number of log messages in a computer system grow fast, and analysing the log data by hand is a time-consuming job. The objective of this study is to propose and implement a scalable architecture for doing real-time log analysis. Log data is structured so that analysis can take place, and the solution is horizontally scalable in every module so that the approach can scale with an ever-growing software solution. The focus of the study is on scalability, and ease-of-use of the implementation of the proposed approach. The proposed solution can scale horizontally and the test set up showed that reporting features for anomalies remained instantaneous when processing 1.2 million log lines per minute. The usability of the proposed approach is tested in a case study at Weave, where bugs were found by running the proposed solution in a controlled environment. ... Read more

This report consummates the Bachelor Project TI3806 course. To accumulate a Bachelor's degree in Computer Science and Engineering at the Delft University of Technology, it is compulsory to pass this course. The project, spanning a duration of ten weeks, is described in this report. The project client was Adviesgroep Strategisch Gebouwbeheer Nederland B.V. (ASG). This is a Dutch energy consultancy company based in Delft that uses big data innovations for energy management and for gaining insight into sustainability issues. The client encountered a real-life problem regarding the transmission of the data from their measuring devices. We were tasked to solve this problem. The project goal was to create a better data transmitter for the client. For naming convention purposes, we call the transmitter a gateway, as the device acts as a port between the measuring devices and the servers. It receives data from the measuring devices within its proximity and after filtering and processing this data, it is transmitted to the servers. The main objective was to design and develop a gateway that is better than the previously used gateway, the one that the client encountered numerous issues with. This improved gateway would then be deployed throughout the country such that the client enjoys numerous improvements. The most important improvements include cost reduction and a more robust, feature-rich and configurable solution.
Furthermore, the goal of this report is to inform the reader about the work that has been completed during the duration of the project at ASG. This report also contains possible future recommendations for this project. The objective of the future recommendations is to act as advice, guidance and instructions for the continuity of this project as the client might decide to further expand the solution by co-workers. ... Read more