Researchers from the University of California, Berkeley have developed a novel method for pruning large transformer models, including those used in vision and language tasks. This technique, framed as a damage-aware multi-armed bandit problem, aims to identify and remove complete functional units within transformers with minimal degradation in performance. The approach involves masking attention heads and MLP channel groups to measure their impact on model loss, then using sampling strategies like Thompson sampling to sequentially select units for removal. Experiments on various datasets and models like GPT-2, Qwen2.5, and ViT-B/16 demonstrated that this bandit-based pruning generally reduces degradation compared to other budgeted methods. AI
IMPACT This research could lead to more efficient deployment of large language and vision models by reducing their size with minimal performance loss.
RANK_REASON The item is an academic paper detailing a new method for pruning transformer models. [lever_c_demoted from research: ic=1 ai=1.0]
- Benjamini–Hochberg procedure
- DeiT-Tiny
- GPT-2
- Imagenette
- lambada
- Open Pre Trained Transformer
- Pythia
- Qwen2.5
- SmolLM2
- Swin-Tiny
- Thompson sampling
- transformers
- University of California, Berkeley
- ViT-B/16
- WikiText-2
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
