Robustness of CNN-based malware byteplot classification under standard image transformations

Abstract

CNN-based malware byteplot classifiers achieve high accuracy under clean conditions, but their robustness to input perturbations remains poorly understood. This paper systematically evaluates how a ResNet18 classifier degrades under standard image transformations (rotations, brightness and contrast shifts, and flips) applied to both grayscale and RGB byteplot representations of a 14-class, 20,020 image dataset. Each transformation is grounded in a realistic attacker model via its correspondence to a binary level obfuscation technique. Results show that rotations are catastrophic even at small angles, flips exhibit strong asymmetry driven by the vertical structure of byteplot sections, and photometric shifts are tolerated until a certain threshold. Training on a mixed set that includes 25% distorted images substantially recovers robustness across all categories, with gains being largely self-attributing to their matching transformation type. RGB representations amplify whatever performance trend is already present in grayscale, for better or worse. Together, the findings reveal that CNN byteplot classifiers exploit specific spatial and photometric properties of their input, and that targeted augmented training can harden them against realistic evasion strategies without sacrificing clean image performance.