New LEAP curriculum boosts Vision Transformer distillation efficiency

Researchers from the University of Oxford have introduced LEAP, a novel training curriculum designed to improve the efficiency of knowledge distillation for Vision Transformers (ViTs). LEAP utilizes a progressive approach, using a teacher model's intermediate features as increasingly difficult targets for the student model. This method accelerates convergence and has shown significant accuracy improvements on datasets like ImageNet-100, with a +12.24% gain for a ViT-S model. Additionally, LEAP reduces training FLOPs by 25.1% and training time by 21% by optimizing teacher inference. AI

IMPACT Enhances efficiency and accuracy in deploying Vision Transformers for edge devices.

RANK_REASON The item is a research paper detailing a new method for model distillation. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• DINOv2
• ImageNet-100
• knowledge distillation
• Leap
• Paris
• University of Oxford
• vision transformer

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →