Consistently Informative Soft-Label Temperature for Knowledge Distillation
Researchers have developed a new knowledge distillation technique called CIST, which addresses the limitations of fixed temperature scaling in transferring knowledge from teacher to student models. CIST assigns separate, sample-wise adaptive temperatures to both models, allowing for more consistent information transfer and relaxing rigid logit-scale alignment. This method has demonstrated consistent improvements on vision and language distillation tasks with minimal computational overhead. AI
IMPACT Improves efficiency of transferring knowledge between AI models, potentially leading to more capable and compact AI systems.