Assessing Robustness of ML-Based Program Analysis Tools using Metamorphic Program Transformations

Assessing Robustness of ML-Based Program Analysis Tools using Metamorphic Program Transformations

Applis, L.H. (TU Delft Software Engineering)
Panichella, A. (TU Delft Software Engineering)
van Deursen, A. (TU Delft Software Technology)

Software Technology

2021

Metamorphic testing is a well-established testing technique that has been successfully applied in various domains, including testing deep learning models to assess their robustness against data noise or malicious input. Currently, metamorphic testing approaches for machine learning (ML) models focused on image processing and object recognition tasks. Hence, these approaches cannot be ap- plied to ML targeting program analysis tasks. In this paper, we extend metamorphic testing approaches for ML models targeting software programs. We present Lampion, a novel testing frame- work that applies (semantics preserving) metamorphic transforma- tions on the test datasets. Lampion produces new code snippets equivalent to the original test set but different in their identifiers or syntactic structure. We evaluate Lampion against CodeBERT, a state-of-the-art ML model for Code-To-Text tasks that creates Javadoc summaries for given Java methods. Our results show that simple transformations significantly impact the target model be- havior, providing additional information on the models reasoning apart from the classic performance metric.

Metamorphic Testing
Machine Learning
Documentation Generation
Code-To-Text
Deep learning

http://resolver.tudelft.nl/uuid:250720e7-34f8-4718-adf4-7191d3c4b48d

IEEE/ACM International Conference on Automated Software Engineering, 2021-11-14 → 2021-11-20, virtual event

Institutional Repository

conference paper

© 2021 L.H. Applis, A. Panichella, A. van Deursen