Safety evaluation of Autonomous Driving Systems (ADSes) requires identifying the conditions under which behaviour transitions from safe to unsafe. However, exhaustive exploration of the scenario parameter space is computationally prohibitive. To address this issue, we introduce a boundary exploration approach inspired by contour-tracing from image processing to efficiently reconstruct the collision boundary in two-dimensional scenario spaces. Our proposed approach operates in three phases: (i) an initialization phase that seeds the search either with randomly sampled scenarios or with scenarios sampled along the boundary of a reference driver model; (ii) a boundary-discovery phase that locates an initial collision/non-collision transition using either a breadth-first alternating search or a binary-search subdivision of the parameter space; and (iii) a boundary-tracing phase that applies Moore-Neighbour Tracing to contour the collision region while avoiding unnecessary sampling. We evaluate four variants of the approach over 35 search spaces of two logical scenarios, using Autoware as ADS, and compare them against exhaustive grid search. Overall, the approach identifies the safety boundary by sampling less than 11% of the search space, yielding nearly a tenfold speed-up with no loss of boundary fidelity. Moreover, it has a minimal overhead in terms of sampled scenarios that are not part of the boundary. ... Read more

Researchers and practitioners have designed and implemented various automated test case generators to support effective software testing. Such generators exist for various languages (e.g., Java, C#, or Python) and various platforms (e.g., desktop, web, or mobile applications). The generators exhibit varying effectiveness and efficiency, depending on the testing goals they aim to satisfy (e.g., unit-testing of libraries versus system-testing of entire applications) and the underlying techniques they implement. In this context, practitioners need to be able to compare different generators to identify the most suited one for their requirements, while researchers seek to identify future research directions. This can be achieved by systematically executing large-scale evaluations of different generators. However, executing such empirical evaluations is not trivial and requires substantial effort to select appropriate benchmarks, setup the evaluation infrastructure, and collect and analyse the results. In this Software Note, we present our JUnit Generation Benchmarking Infrastructure (JUGE) supporting generators (search-based, random-based, symbolic execution, etc.) seeking to automate the production of unit tests for various purposes (validation, regression testing, fault localization, etc.). The primary goal is to reduce the overall benchmarking effort, ease the comparison of several generators, and enhance the knowledge transfer between academia and industry by standardizing the evaluation and comparison process. Since 2013, several editions of a unit testing tool competition, co-located with the Search-Based Software Testing Workshop, have taken place where JUGE was used and evolved. As a result, an increasing amount of tools (over 10) from academia and industry have been evaluated on JUGE, matured over the years, and allowed the identification of future research directions. Based on the experience gained from the competitions, we discuss the expected impact of JUGE in improving the knowledge transfer on tools and approaches for test generation between academia and industry. Indeed, the JUGE infrastructure demonstrated an implementation design that is flexible enough to enable the integration of additional unit test generation tools, which is practical for developers and allows researchers to experiment with new and advanced unit testing tools and approaches. ... Read more

We report on the results of the eighth edition of the Java unit testing tool competition. This year, two tools, EvoSuite and Randoop, were executed on a benchmark with (i) new classes under test, selected from open-source software projects, and (ii) the set of classes from one project considered in the previous edition. We relied on an updated infrastructure for the execution of the different tools and the subsequent coverage and mutation analysis based on Docker containers. We considered two different time budgets for test case generation: one an three minutes. This paper describes our method- ology and statistical analysis of the results, presents the results achieved by the contestant tools and highlights the challenges we faced during the competition. ... Read more