A Generic and Automatic Test Strategy for Compiler Testing

Abstract

Domain-specific Languages (DSLs) are languages specifically tailored for an application or expert domain. These can be implemented as compilers, which check the correctness of an input program and translates it to a target language. Manual testing of compilers is a time consuming and labor intensive task. This motivates the development of approaches to facilitate the quality assurance process. In this thesis we present an automatic and generic test strategy for the generation of test cases for Spoofax developed compilers. We use a program generator to generate large syntactically correct programs from Syntax Definition Formalism (SDF) grammars. Additionally, we improve the program generator with an expansion of our generation algorithm to use Name Binding Language (NaBL) modules to generate partial name correct programs. We also provide a DSL to define error fixes that are used to attempt the repair of static semantic errors reported after compilation. After program generation we use a partial oracle to automatically detect failures during the invocation of the compiler. Finally, we provide a heuristic to reduce the size of generated programs, whilst preserving their failure inducing behavior. This test strategy was used to generate test cases forWebDSL, a DSL targeting the domain of developing dynamic web applications with a rich data model. The generated test cases unveiled eleven unique faults in the analysis phase of compilation. These were reported together with the programs reduced by our program shrinking heuristic and they were positively received by the WebDSL development team.