Metamorphic-Based Many-Objective Distillation of LLMs for Code-related Tasks

Abstract

Knowledge distillation compresses large language models (LLMs) into more compact and efficient versions that achieve similar accuracy on code-related tasks. However, as we demonstrate in this study, compressed models are four times less robust than the original LLMs when evaluated with metamorphic code. They exhibit a 440% higher probability of misclassifying code clones due to minor changes in the code fragment under analysis, such as replacing parameter names with synonyms. To address this issue, we propose Morph, a novel method that combines metamorphic testing with many-objective optimization for a robust distillation of LLMs for code. Morph efficiently explores the models' configuration space and generates Paretooptimal models that effectively balance accuracy, efficiency, and robustness to metamorphic code. Metamorphic testing measures robustness as the number of code fragments for which a model incorrectly makes different predictions between the original and their equivalent metamorphic variants (prediction flips). We evaluate Morph on two tasks-code clone and vulnerability detection-targeting CodeBERT and GraphCodeBERT for distillation. Our comparison includes Morph, the state-of-theart distillation method AVATAR, and the fine-tuned non-distilled LLMs. Compared to Avatar, Morph produces compressed models that are (i) 47% more robust, (ii) 25% more efficient (fewer floating-point operations), while maintaining (iii) equal or higher accuracy (up to +6%), and (iv) similar model size.