Vision Transformer Finetuning Benefits from Non-Smooth Components
A new research paper published on arXiv explores the concept of "plasticity" in Vision Transformers, defining it as the average rate of change within model components. The study suggests that prioritizing components with high plasticity, such as attention modules and feedforward layers, leads to improved finetuning performance. This finding challenges the conventional wisdom that smoothness is always beneficial for transformer models, offering a novel perspective on their functional properties. AI
IMPACT Challenges conventional assumptions about transformer smoothness, potentially guiding future model adaptation strategies.