Distilling Knowledge for Assertion Generation: Alpha-Temperature Tuning in Smaller Language Models

Abstract

Testing of software is crucial to the quality of the final product manual test assertion creation has become a significant bottleneck in the development process, which delays release. Having shown promise in generating assertions automatically, Large language models (LLMs) have showed promise in generating assertions automatically. This is due to their fluency in both natural languages and code, as well as the fact that they produce tests a lot faster than a developer would. However, LLMs must reckon with deployment issues that come with the high computation time and latency of large models, or the limited functionality of their smaller, locally-executable counterparts. Knowledge distillation, a technique that aims to "transfer knowledge" from a teacher model to a student one, can thus enable the potential of smaller and faster models. This drives the research to explore the effectiveness of knowledge distillation in developing a smaller and efficient model for assertion generation. With CodeT5 as the teacher model, the student model learns from the teacher. The student is iteratively trained in epochs, validated on unseen data. The metrics used to evaluate include assertion accuracy, similarity to teacher model output and ground truth, model size, inference time, with the goal to quantify the trade-offs and determine the feasibility of distilled models for practical assertion generation. We presented and analyzed the results we achieved. The capability the student showed was around 1/3 of that of the teacher, which suggest a potential for creating efficient, yet reliable assertion generation tools.