The continual increase in cyber crime revolving blockchain applications calls for secure blockchain systems and clarity on the division of security responsibilities. This research is an integrated project between two master programmes at the Delft University of Technology: Computer Science and Communication Design for Innovation, and focuses on software testing and security responsibilities. In this study, we investigate if grammar-based fuzzing, a popular approach for identifying bugs in software, is effective on JSON-RPC systems like blockchain applications Ripple and Ethereum. Furthermore, we evaluate whether we can improve upon traditional grammar-based fuzzing by using evolutionary search. We introduce GEFRA, a black-box grammar-based fuzzing tool that generates tests for JSON-RPC APIs. Using a diversity-based fitness function that leverages system feedback, GEFRA is able to effectively guide the search process towards new test cases that obtain additional test coverage. Additionally, various perspectives on blockchain security responsibilities are investigated. A media content analysis was performed and interviews were conducted with legal and blockchain experts. News media frequently frame end users as responsible for the prevention of blockchain attacks. While attackers are legally responsible, users are left to deal with the consequences if attackers cannot be found. Responsibilities generally end up with users as decentralisation is the core idea of blockchain. Legislation may be the only solution to define a clear division of responsibilities.

On the intuitive level, software testing is important because it assures the quality of the software used by humans. However, ensuring this quality is not an easy task because as the complexity of the software increases, so do the efforts to test it. Search-based software testing is an active research field that develops and explores tools for automatic test case generation. Their work involves using meta-heuristic search optimisation approaches such as evolutionary algorithms from the evolutionary computation community and applying them to test case generation. The crossover between the evolutionary computation domain and the test case generation one produced DynaMOSA, a state-of-the-art evolutionary algorithm for generating test cases. In an attempt to produce another well-performing algorithm, this paper performs another crossover between the two communities and creates DynaMOSARVEA - a product of a Reference Vector Guided Evolutionary Algorithm (RVEA) and DynaMOSA. The conducted empirical study showed that although DynaMOSARVEA does not outperform DynaMOSA, it did outperform RVEA, thus demonstrating the value brought by domain-specific knowledge.

The XRP network (Ripple network) is a global transaction network that settles trans- actions in seconds. It is a technology that provides new opportunities for traditional fi- nancial institutions and startups. In the XRP network, participants depend on each other’s functioning. However, at present Ripple Labs, Inc. is the only actor monitor- ing some form of performance. This paper presents a way for independent individuals to verify the score of validators. Further- more, to improve the insight into the per- formance of validators, a new metric is in- troduced. Together these improvements con- tribute towards a healthy decentralised XRP network.

Ripple is a relatively new payments network that aims to improve the financial system by unifying its underlying infrastructure. Given its critical function, its system must be reliable and free of bugs. Therefore it should be tested extensively. One of the test methods that has not been used on it yet is log inference, a method that has a good potential for modelling complex communication protocols. Therefore, we have developed an empirical model of the Ripple Consensus Protocol by learning a Deterministic Finite Automaton (DFA) from the log files of two servers in the Ripple network. We have also developed a theoretical DFA of the Ripple Consensus Protocol and compared this to the empirical model to verify that the two systems function comparably. There has been found one notable difference between the two models, but whether this difference has a critical impact remains to be discussed.

Understanding the behaviour of a software system plays a key role in its development and maintenance processes. Unfortunately, accurate and concise models are not always available during development, due to the rapid changes in the structure and scale systems may undergo during this phase. Finite State Machines (FSM) are a natural and prevalent template for modeling system behaviour, and inferring such models through profiling, source code analysis, and log analysis has been an active area of research over the last decades. However, the applicability of existing techniques is restricted by different factors: profiling introduces significant overhead, which is infeasible for many real-time systems; the source code of a system may not always be available for analysis; and current log analysis approaches do not scale well with the number of logs produced by the system. In this paper, four meta-heuristic search approaches are used to create FSM models for the XRP Ledger consensus algorithm from its logs, addressing the scalability problem by approximating a FSM while only exploring a small portion of the search space. Random Selection Stochastic Hill Climber (RSSHC), Tournament Selection Stochastic Hill Climber (TSSHC), Simulated Annealing (SA), and Pareto Simulated Annealing (PSA) are considered. The performance of the model produced by each algorithm is quantified using an accuracy-based evaluation technique, and the accuracy, conciseness, and runtime of the approaches are compared. The results indicate that PSA produces the most accurate and least concise models with a consistent accuracy of over 92%. RSSHC and TSSHC obtain solutions with a slightly lower accuracy, but significantly better conciseness, while SA produces the most compact and least accurate models. All four algorithms produce results within minutes and scale linearly with the size of the problem.

Software testing is an important but time-consuming task, making automatic test case generation an appealing solution. The current state-of-the-art algorithm for test case generation is DynaMOSA, which is an improvement of NSGA-II that applies domain knowledge to make it more suitable for test case generation. Although these enhancements are applicable to other evolutionary algorithms, no research has been done on how effective other algorithms can function as the base. In this paper, we apply the DynaMOSA modifications to SPEA-II to create a new algorithm, DynaSPEA-II. We conduct an empirical experiment where we evaluate the DynaMOSA enhancements, and directly compare DynaSPEA-II to DynaMOSA. The algorithms are assessed on a benchmark consisting of 36 diverse JavaScript classes w.r.t. branch coverage. Our results show that adding DynaMOSA enhancements to SPEA-II results in higher coverage in 13.9% of classes, with an average increase of 4.92% for classes where a statistically significant difference was found. DynaSPEA-II performed equally to DynaMOSA, with no statistically significant difference being found between the two.

Software testing, a critical phase in the software development lifecycle, is often hindered by the time-intensive and costly manual creation of test cases. While automating test case generation could mitigate these challenges, its adoption in the industry has been limited due to difficulties in comprehending the generated test cases. To address this, our study presents an approach for clustering test cases and evaluates its impact on the comprehensibility of test suites through empirical research. Our approach clusters test cases based on their covered objectives, grouping together those with similar attributes to enhance developer understanding. The core of our empirical research evaluates developer agreement with our clustering method and contrasts the comprehensibility of clustered versus non-clustered test suites. Findings suggest a broad agreement among developers in favor of our clustering approach, with clustered test suites facilitating faster software maintenance tasks. Notably, the effectiveness of task completion remained comparable between both suite types. In summary, our research introduces and validates an innovative test case clustering strategy, striving to enhance the comprehensibility of automatically generated test suites.

Traditional software testing is a labor-intensive and expensive manual process. To mitigate the high cost of manual test case generation, researchers have developed various techniques for automated test case generation over the last few decades. These techniques make use of static type information to determine which data types should be used in new test cases. Dynamically typed languages like JavaScript do not provide type information. The lack of type information poses a new challenge for test case generation techniques. In this thesis, we propose a novel unsupervised probabilistic type inference approach to infer data types in a test case generation context. The approach uses both static and dynamic type inference techniques. We implemented the approach in a novel tool called SYNTEST-JAVASCRIPT which is an extension of the SYNTEST-FRAMEWORK. We evaluate the performance of the approach compared to random type sampling with respect to branch coverage. The evaluation is done using a custom benchmark of 97 units under test. Our results show that using statically inferred type achieves a statistically significant increase in 54% of the benchmark files compared to the baseline. The combination of using both statically and dynamically inferred types improves the approach slightly with a significant increase in 56% of the benchmark files compared to the baseline. Finally, the results show that the time consumed by the static and dynamic type inference is insignificant compared to the total time budget and is worthwhile given the performance boost type inference provides.

To validate the quality of software, test cases are used. These test cases are often manually-written, which is labor-intensive. To avoid this problem, automated software testing was invented. Search-based software testing is a useful tool for developers to automatically generate test cases. However, improvements are still needed to create test cases that compete with manually-written ones. EvoMaster is a tool that generates system-level test cases for RESTful APIs using the MIO algorithm. An important aspect of this algorithm is sampling new test cases. Currently, EvoMaster employs random and smart sampling to achieve this goal. This paper aims to improve the coverage of the generated tests by expanding the sampling methods with seeded sampling. This method consists of extracting sequences of HTTP requests from manually-written tests and using these to sample new test cases. Seeded sampling is evaluated on two RESTful APIs with 7 different parameter sets. We show that the addition of seeded sampling can improve the coverage achieved by EvoMaster compared to the current combination of sampling techniques. Nonetheless, this paper has some limitations. It only takes two RESTful APIs into account and has a small amount of benchmark runs to back its findings.

Software testing is an important yet time consuming task in the software development life cycle. Artificial Intelligence (AI) algorithms have been used to automate this task and have proven to be proficient at it. This research focuses on the automated testing of JavaScript programs, and builds upon the existing SynTest framework that is the current state of the art, with the Dynamic Many Objective Sorting Algorithm (DynaMOSA) being the best performing AI algorithm for test case generation. DynaMOSA uses the Non-Dominated Sorting Algorithm - II (NSGA-II) as its base algorithm, and adds modifications to it. This paper investigates whether the use of the Pareto Envelope Based Search Algorithm - II (PESA-II) as the base algorithm results in improved performance. The contributions of this research includes a modified PESA-II integrated into the SynTest framework, using inspiration from DynaMOSA. Moreover, we answer the question "How does the modified PESA- II perform compared to DynaMOSA in generating test cases for JavaScript programs?" The performance of the algorithms is measured based on the (branch and method) coverage of the test cases generated for a suite of JavaScript classes. The results show that the modified version of PESA-II outperforms the base version. However, neither manage to outperform DynaMOSA.

Automated generation of system tests for RESTful APIs has been extensively investigated. Previous investigations use either a white box or a blackbox approach, wherein the quality of the test cases can be assessed on the HTTP response in the prior and also on the results of byte-code analysis in the latter. Both approaches are limited however, as the black box is often under performing, while the white box can only be applied to a limited set of RESTful APIs. In this paper, we introduce a novel approach where the system under test (SUT) is defined as a container. In this approach, the log output can be used to assess the quality of the test cases. We present a prototype that retrieves all semantically relevant in-formation from the logs using regex patterns, which was subsequently maximized by an evolutionary algorithm. The container, white box and black box mode were performed on three SUTs to evaluate the effectiveness and performance of these modes. An increase in code coverage was observed in the container versus black box mode in all SUTs (p <0.001,p <0.001 and p= 0.054). In all SUTs, significantly less actions could be evaluated by the container as compared to the black box and white box mode. Our results show promising results for the novel approach outlined in this paper. Importantly, this approach can be applied to any RESTful API

Randomized variational operators can be very disruptive to the search process, especially when there exist dependencies between the variables under search. Within test-cases, these dependencies exist as well. This makes it interesting to evaluate the benefits of preserving these dependencies during test-case generation. In this paper, we propose two variants of the Many-Objective Sorting Algorithm (MOSA). The first of which is based on Agglomerative Clustering, ACMOSA. The second is a Gene-pool Optimal Mixing based variant, GOMOSA. ACMOSA and GOMOSA model the inter-gene dependencies and use that model to intelligently perform crossover while preserving key building blocks within individuals. These novel techniques are evaluated in an empirical study and compared to MOSA and the Many Independent Objective algorithm (MIO). This study is composed of several benchmark RESTful APIs for which the algorithms generate test-cases. The results of the empirical study show that, for 40% of the tested APIs, the novel techniques provide a significant benefit time-wise. For another 40% of the APIs, they perform equally well, and for 20% of the APIs under evaluation they performed worse.

An essential step of software development is obtaining an understanding of the behaviour of a system. Accurate state models of system behaviour might help software developers build such an understanding. There exist several techniques for automatically inferring models on system behaviour using log analysis, but these do not scale well for systems that produce large amounts of logs. In this study, we present an approach for inferring concise state models of system behaviour using log analysis, called MASM (Markov Algorithm for inferring concise State Models), which implements a Markov chain. We argue this approach has the potential for higher scalability than existing techniques, as it attempts to approximate a state model by exploiting the properties of Markov chains. This is achieved by considering a sequence of log statements as a stochastic process, which enables MASM to infer a naive state model from a set of log statements, which is then minimized using the properties of Markov chains. We evaluated MASM in an empirical study on the XRP Ledger Consensus Protocol. In this empirical study, MASM was evaluated on accuracy at different compression rates, and scalability. The results indicate that MASM shows several knee points in terms of accuracy, suggesting several optimal compression rates. We also found that the run-time of MASM scales linearly with the size of the dataset. Finally, we discovered that MASM is unable to outperform a random compression algorithm, which compresses the model by randomly merging states, in terms of accuracy. However, further research is required on measuring the readability of the produced models to derive definitive conclusions on the performance of MASM compared to a random compression algorithm.

In recent decades, automatic test generation has advanced significantly, providing developers with time-saving benefits and facilitating software debugging. While most research in this field focused on search-based test generation tools for statically-typed languages, only a few have been adapted for dynamically-typed languages. The larger search-space, generated by the dynamic allocation of types, causes standard search-based algorithms to not be as efficient in this domain and requiring a different approach. Existing algorithms like NSGA-II, MOSA and DynaMOSA have been employed to address this problem, but exploring different approaches may yield better results. That is why this paper proposes a different procedure based on an adaptation of the particle swarm optimization algorithm (PSO). The adaptation was evaluated using the SynTest framework, showing that DynaMOSA achieves better results than the presented approach, both when comparing the PSO adaptation with and without DynaMOSA features, and when comparing the base DynaMOSA algorithm with PSO adapted to include DynaMOSA ingredients.

Large software systems today require increasingly complex models of their execution to aid the analysis of their behavior. Such execution models are impractical to compile by hand, and current approaches to their automated generation are either not generalizable or not scalable enough. This paper addresses this problem with a new approach based on the interpretation of log traces. We analyze the effectiveness of using community detection algorithms for generating system execution models from structured datasets of log samples. This approach first models sets of log traces as tree-shaped automata, and then uses graph clustering algorithms to reduce such tree representations down to more concise models. This research focuses on analysing the quality of the generated models in terms of conciseness, accuracy, recall and scalability. Testing was performed on data samples from the XRP network, a blockchain-based payment system. During implementation of a proof of concept, multiple challenges arose which limited the ability of our study to fully evaluate the approach's effectiveness. The partial results obtained show poor performance, both in terms of runtime and in accuracy of generated models. Due to the limitations of the evaluation performed, the results are to be considered exploratory and require further testing.

System behavior models are highly useful for the developers of the system as they aid in system comprehension, documentation, and testing. Even though methods to obtain such models exist, e.g. profiling, tracing, source code inference and existing log-based inference methods, they can not successfully be applied to the case of large, real-time systems. Profiling and tracing add overhead which may alter the system's behavior and source code inference does not scale for systems of this magnitude. Existing log-based approaches also suffer due to the intrinsic scalability issues of deriving a minimal model as proven by Gold. In this work, this issue is tackled by applying Reinforcement Learning to the model inference problem. First, an initial model is created from the traces and then Q-Learning is applied to shrink this model into a concise and accurate representation of the system. The approach is evaluated using log traces produced by the XRP Ledger Consensus Protocol. Its effectiveness is assessed based on the accuracy and the conciseness of the inferred models as well as the execution time of the algorithm to infer the model. Results show that the Q-Learning implementation used in this work, is not able to converge to consistent action values. These results might be implementation specific meaning future work should experiment with and extend the current implementation of the algorithm, or due to assumptions made in this work about the underlying systems do not hold. Future work should apply this approach to a different system so as to assess its feasibility.

Kotlin is a programming language best known for its interoperability with Java, as well as the measurable improvements it offers over it. Since it became Android’s go-to language in 2019, the popularity and impact of Kotlin have risen greatly. Amidst this surge in popularity, the Kotlin developer team is working on a new version of the compiler that introduces sweeping changes to the ecosystem. Traditional compiler testing is a manual and laborious task that requires extensive developer effort and expertise. In an attempt to mitigate this, researchers have invested great resources in developing and perfecting automated compiler testing tools over the last decades. These approaches generate new pieces of code to test the behavior of compilers, which is assessed through differential testing. However, the usage of heuristics as guidance for the generative process is not well understood, and no approach that generates Kotlin code from scratch currently exists. In this thesis, we propose a novel method of enriching standard grammar specifications with language-targeted semantic context that is integrated in the sampling process. We structure generated code hierarchically and use it as the base of an evolutionary computation framework. Within this framework, we introduce two classes of algorithms that are novel to the field of compiler fuzzing, based on syntactic diversity and semantic proximity, respectively. We carry out an empirical analysis spanning 200K generated Kotlin files, which we analyzed through different Kotlin compiler versions. Our results uncovered five previously unreported categories of bugs, which we reported to the Kotlin compiler developer team. The developers verified and replicated our instances on the current re- lease of the Kotlin compiler, and have assigned target release dates for fixes within the current major version of the compiler. The study also provides new insight into the effects of heuristic-specific hyperparameters such as expression simplicity, dissimilarity measurements, and target selection.