Analysis and Tuning of Knowledge Distillation for Efficient Collaborative Learning

Abstract

Knowledge Distillation (KD) has become a crucial technique for efficient collaborative learning in distributed systems, especially under data and system heterogeneity. Despite considerable attention, few works ystematically examine tuning strategies for KD hyperparameters or provide comprehensive empirical comparisons across different data distributions. In this paper, we conduct an in-depth study of multiple KD methods—Vanilla KD, Deep Mutual Learning (DML), Data Partitioning KD (DP-KD), and our Tuned KD approach—using diverse data partitions and transfer sets. We show that (1) hyperparameter tuning dramatically boosts KD performance, especially when teacher–student models have disparate accuracies, (2) different transfer set sizes and labeling conditions substantially affect outcomes. Our findings serve as practical guidelines for effectively applying KD to collaborative learning tasks.