New iBKD framework boosts Vision Transformer performance in low-data settings

Researchers have developed a new distillation framework called iBKD to improve the performance of Vision Transformers (ViTs) when training data is limited. Unlike general knowledge distillation methods that discard spatial information, iBKD preserves the grid structure throughout the transfer process. This is achieved through an Inductive Bias Attention Module that aggregates student layers onto the teacher grid, sharpens structural cues, and injects them via convolutional cross-attention. The iBKD framework only requires training, leaving the deployed ViT model unmodified and without inference overhead. Experiments across various ViT backbones and data-scarce benchmarks show iBKD outperforming existing methods, with its effectiveness increasing as training data decreases. AI

IMPACT This research offers a method to improve the efficiency of Vision Transformers in data-scarce environments, potentially reducing the need for massive datasets in certain applications.

RANK_REASON The item describes a novel research paper proposing a new method for knowledge distillation in machine learning. [lever_c_demoted from research: ic=1 ai=1.0]

Read on Hugging Face Daily Papers →

• CNN
• iBKD
• Inductive Bias Attention Module
• Vision Transformers
• Vít

• paper
• model release

AI-generated summary · Google Gemini · from 2 sources. How we write summaries →