Software applications inevitably crash, and it is time-consuming to recreate the crash conditions for debugging. Recently, researchers have developed frameworks relying on genetic algorithms, e.g. Botsing, for automated crash reproduction. However, the existing approaches process exceptions of different types as if they were the same. In this thesis, we study how the four most common types of Java exceptions are thrown and define specialised fitness functions for them. We have extended Botsing and carried out an evaluation against 52 real-world crashes from seven various open-source software applications. Our results show that our proposed fitness functions influence both the effectiveness and efficiency, negatively or positively depending on the type of the target exception. This thesis demonstrates how tailoring the fitness functions according to the exception type can improve search-based crash reproduction.

To ensure that a software system operates in the correct way, it is crucial to test it extensively. Manual software testing is severely time-consuming, and developers often underestimate its importance. Consequently, many tools for automatic test generation have been developed during the past decade. EvoSuite is a state-of-the-art tool for automatic generation of unit tests. It can produce test suites based on chosen coverage criteria, also known as a fitness function. Previous studies have widely assessed the performance of the different fitness functions available in EvoSuite. However, the combination of various coverage criteria has not been considered. In this paper, we assess the effectiveness of the combination of Branch coverage and Output diversity fitness functions. We compare it to two of the most popular fitness functions in EvoSuite - Branch coverage and the Default configuration (combines eight coverage criteria) to estimate its performance. We developed a machine learning tool that determines which fitness function will achieve better results based on class characteristics. The assessment criteria we consider are branch coverage and fault detection, represented by mutation score. We further examined how the time limit affects the performance of the considered fitness functions. The results have shown that the combination of Branch coverage and Output diversity outperforms the Default configuration significantly in branch coverage but has worse performance in fault detection capabilities. We have also found that the Branch and Output diversity coverage criteria achieve better results when compared with only using Branch coverage in terms of mutation score. Additionally, the static software metrics, especially CBO, LCOM* and LOC, are highly correlated with the performance of the fitness functions and can predict which coverage criteria will achieve better results.

Various search-based test generation techniques have been proposed to automate the process of test generation to fulfill different criteria (e.g., line coverage, branch coverage, mutation score, etc.). Despite these techniques' undeniable accomplishments, they still suffer from a lack of guidance coming from the data gathered from the production phase, which makes the generation of complex test cases harder for the search process. Hence, previous studies introduced many strategies (such as dynamic symbolic execution or seeding) to address this issue. However, the test cases created by these techniques cannot assure the full coverage of the execution paths in software under test. Therefore, this thesis introduces common and uncommon behavior test generation (CUBTG) for search-based unit test generation. CUBTG uses the concept of commonality score, which is a measure of how close an execution path of a generated test case is from reproducing the same common and uncommon execution patterns observed during the real-world usage of the software. To evaluate the performance of CUBTG, we implemented it in EvoSuite and evaluated it on 150 classes from JabRef, an open-source application for managing bibliography references. We found that CUBTG managed to cover more common behaviors than plain MOSA in 75% of the cases, and more uncommon behaviors in 60% of the cases. In up to 10% of the cases CUBTG managed to find more mutants seeded by PIT by using method sequences that plain MOSA did not find.

Cyber-physical systems are complex systems constructed from different independent parts. A self-driving car is an example of a cyber-physical system where independent parts have to come together in order to result in a car that is able to drive by itself. The main challenge is finding failures within the interactions between the independent parts of the self-driving system. In this paper, we present a novel algorithm REWOSA, in order to detect these faults within the self-driving software. With the use of real-world roads extracted from Google Maps combined with a multi-objective genetic algorithm, we develop a new way to generate roads for testing self-driving cars. We evaluate this algorithm against a state-of-the-art multi-objective genetic algorithm using randomly generated roads using two different setups. Our results show that REWOSA is able to generate more failures than the baseline on both the setups, as well as create more complex roads. In return, REWOSA does create a large overhead due to the complexity of the real-world roads. However, this overhead is justifiable as we can detect more faults with the more complex real-world roads.

Debugging application crashes is an expensive and time-taking process, relying on the developer’s expertise, and requiring knowledge about the system. Over the years, the research community has developed several automated approaches to ease debugging. Among those approaches, search-based crash reproduction, which tries to generate a test case capable of reproducing a given crash to make it observable to the developers, solely based on the stack trace included in the crash report. We believe that this makes crash reproduction the perfect candidate to achieve end-to-end crash fault localization. In this thesis, we explore and empirically evaluate the usage of search-based crash reproduction combined with spectrum-based fault localization on 50 real-world crashes. Starting from a crash report, we generate crash-reproducing test cases and use them in conjunction with the existing or an automatically generated unit test suite as input for spectrum-based fault localization. Our results show that, although, hand-written test cases remain the most efficient in the general scenario, automatically generated crash-reproducing test cases still reduce the number of statements to be investigated by developers. Additionally, when considering the best-case scenario where only crash-reproducing test cases covering the fault are evaluated, we observe no statistically significant difference between the accuracy of fault localization when using hand-written or automatically generated test cases. Our results confirm the feasibility of end-to-end automated crash fault localization. The results also identify new challenges for both automated test case generation and fault localization, as well as when they are combined.