Kabir Thayani

AI research reveals distillation bottleneck, label-aware methods improve performance

Researchers have identified a significant geometric bottleneck in knowledge distillation between Vision Transformers and smaller CNNs. Standard cosine distillation causes the learned representations to collapse to a low…