Guiding automated system test generation for RESTful APIs using log statements

Abstract

Automated generation of system tests for RESTful APIs has been extensively investigated. Previous investigations use either a white box or a blackbox approach, wherein the quality of the test cases can be assessed on the HTTP response in the prior and also on the results of byte-code analysis in the latter. Both approaches are limited however, as the black box is often under performing, while the white box can only be applied to a limited set of RESTful APIs. In this paper, we introduce a novel approach where the system under test (SUT) is defined as a container. In this approach, the log output can be used to assess the quality of the test cases. We present a prototype that retrieves all semantically relevant in-formation from the logs using regex patterns, which was subsequently maximized by an evolutionary algorithm. The container, white box and black box mode were performed on three SUTs to evaluate the effectiveness and performance of these modes. An increase in code coverage was observed in the container versus black box mode in all SUTs (p <0.001,p <0.001 and p= 0.054). In all SUTs, significantly less actions could be evaluated by the container as compared to the black box and white box mode. Our results show promising results for the novel approach outlined in this paper. Importantly, this approach can be applied to any RESTful API