Consistently informative soft-label temperature for knowledge distillation
Submitted to the International Conference on Artificial Intelligence and Statistics (AISTATS 2027)
Abstract
Knowledge distillation trains a small student network to match the softened predictions of a large teacher. A single fixed temperature treats all samples alike, so some soft labels are nearly one-hot while others are almost uniform, and teacher and student are forced onto the same logit scale. We propose CIST, which assigns adaptive per-sample temperatures to teacher and student separately and reweights the distillation loss by teacher confidence and student difficulty. We show that the entropy of a teacher’s soft label is governed mainly by the ratio of its largest logit to the temperature, which motivates the design. CIST improves standard distillation and strong baselines on image classification (CIFAR-100, ImageNet) and language-model distillation, at negligible extra cost.
Cite
Hoang-Chau Luong, Nghia V. Vo, Kaiqi Zhao, and Lingwei Chen. Consistently informative soft-label temperature for knowledge distillation. Submitted to AISTATS 2027. arXiv:2605.20357.