Elastic ViTs from Pretrained Models without Retraining
Researchers have developed SnapViT, a novel method for creating elastic Vision Transformers (ViTs) that can adapt to various computational budgets without requiring retraining. This post-pretraining structured pruning technique efficiently combines gradient information with cross-network structure correlations, approximated via an evolutionary algorithm. Experiments on several pretrained models show SnapViT outperforms existing methods across different sparsities, generating adjustable models in under five minutes on a single A100 GPU. AI
IMPACT Enables more flexible deployment of vision models across diverse hardware constraints.