Inducing Spatial Locality in Vision Transformers through the Training Protocol
Researchers have found that specific training techniques can encourage spatial locality in Vision Transformers. By using a 'Modern' protocol involving data augmentation like CutMix and ColorJitter, along with label smoothing, early layers of ViTs showed more concentrated attention patterns. An ablation study revealed that CutMix was the primary driver of this effect, significantly reducing the Mean Attention Distance compared to baseline methods. AI
IMPACT Training protocols like CutMix can improve the efficiency and interpretability of Vision Transformers by promoting localized attention.