New pruning method significantly reduces Vision Transformer computational costs

Researchers have developed a new method called Denoised Variance-Based Pruning with Optimal Brain Bias Compensation (DVBP + OB$^2$C) to reduce the computational overhead of Vision Transformers (ViTs). This technique uses random matrix theory to filter noise from activation covariance for more robust neuron selection. DVBP + OB$^2$C also optimizes remaining weights using the same statistics gathered for selection, achieving state-of-the-art training-free performance. Experiments show it retains over 90% of original accuracy at 50% MLP pruning on various architectures, significantly outperforming previous methods. AI

IMPACT This method could enable more efficient deployment of large Vision Transformer models on edge devices.

RANK_REASON The cluster contains a research paper detailing a new method for model pruning. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• arXiv
• ConvNeXt
• Data Efficient Image Transformers
• DVBP + OB$^2$C
• Optimal Brain Compression
• Swin Transformer
• Variance-Based Pruning
• Vision Transformers

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →