CutMix training protocol induces spatial locality in Vision Transformers

Researchers have found that specific training techniques can encourage spatial locality in Vision Transformers. By using a 'Modern' protocol involving data augmentation like CutMix and ColorJitter, along with label smoothing, early layers of ViTs showed more concentrated attention patterns. An ablation study revealed that CutMix was the primary driver of this effect, significantly reducing the Mean Attention Distance compared to baseline methods. AI

IMPACT Training protocols like CutMix can improve the efficiency and interpretability of Vision Transformers by promoting localized attention.

RANK_REASON The cluster contains an academic paper detailing a new finding in machine learning model training. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv stat.ML →

• CIFAR-10
• CIFAR-100
• ColorJitter
• CutMix
• Label Smoothing
• Tiny-ImageNet
• Vision Transformer

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →