Multi-Level Contrastive Knowledge Distillation for Resource-efficient Visual Inference on Edge Devices

Abstract

Deploying deep neural networks on networked edge devices enables low-latency inference without cloud dependency, yet is constrained by tight computational, memory, and energy budgets. Knowledge distillation is a widely used approach to compress large models into compact ones suitable for edge hardware, but conventional methods require labeled data that is often costly or unavailable in practical sensor network applications. This paper proposes Multi-Level Contrastive Knowledge Distillation (MLCKD), an unsupervised framework that transfers multi-level relational knowledge from a contrastively pretrained deep teacher model to a lightweight student using only unlabeled data. Unlike existing self-supervised distillation methods that operate solely on the teacher’s final embedding, MLCKD attaches lightweight auxiliary branches at multiple network stages to distill pairwise similarity structures at both intermediate and final representation levels. We introduce a two-stage offline training procedure which first equips the frozen teacher with contrastively trained auxiliary branches, and then optimizes the student to reproduce the teacher’s relational structure at every level via Kullback–Leibler divergence. Evaluation with a ResNet-50-based teacher network and ResNet-18-based student network demonstrates that MLCKD substantially improves top-1 linear evaluation accuracy over the SimCLR baseline and nearly closes the performance gap with the full-capacity teacher. Transfer learning evaluation on eight downstream classification tasks confirms that the distilled representations generalize broadly, with MLCKD outperforming the self-supervised baseline on all tasks and surpassing supervised training by a considerable margin. Inference benchmarks on three edge platforms further show that the student achieves 1.6× to 2.8× speedup over the teacher while delivering stronger representations, confirming that MLCKD enables practical, label-free deployment of resource-efficient visual intelligence on edge hardware.