Logit KD

Study compares feature-based vs. logit-based knowledge distillation

A new study on arXiv investigates knowledge distillation techniques, specifically comparing feature-based methods against logit-based distillation across different student model architectures. Researchers found that whi…