Towards Understanding How Developers Comprehend Tests

Abstract

Developers spend the majority of their time and effort on reading and comprehending source code. In order to improve this process of program comprehension for developers, a numerous amount of existing studies have looked into understanding how developers approach it and factors of influence. However, less is known in the field about how developers comprehend test code, an alternative form of source code, despite its widely acknowledged importance and benefits in both research and practise. In this paper, we perform a foundational study on understanding how developers comprehend tests by applying existing knowledge and work on program comprehension onto tests comprehension and looking at the influential factors. An online controlled experiment was conducted with 44 developers to measure three defined metrics of tests comprehensibility, namely the total time spent on reading a test suite, the ability to identify the overall purpose a test suite, and the ability to produce additional test cases to extend a test suite. The main findings of our study, with several implications for future research and real world, are that (i) prior knowledge of the software project decreases the total reading time, (ii) developers with more Java experience spend proportionally less  time on the Arrange and Assert sections of test cases, (iii) experience with Java and prior knowledge of the software project positively influences the ability to produce additional test cases of certain categories, and (iv) the most influential factor towards understanding and extending a test suite is experience with using tests.