A new study published on arXiv investigates the transfer of attention mechanisms in Vision Transformers (ViTs). Researchers found that while ViTs trained to mimic a teacher model's attention maps achieve high in-distribution accuracy, they fall short under distribution shift. The study indicates that the attention transfer is nearly perfect and permanent, but the observed robustness gap is largely an artifact of training maturity, closing when training schedules are completed. The findings suggest that attention maps in this context reveal where a model looks, rather than what it knows. AI
IMPACT Suggests that attention maps in Vision Transformers may not fully capture a model's knowledge, impacting how we evaluate and improve their robustness.
RANK_REASON Academic paper detailing research findings on Vision Transformer attention mechanisms. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
