SHUFFLESPARSE learned permutations boost structured sparse network accuracy

Researchers have developed SHUFFLESPARSE, a novel permutation primitive designed to enhance structured weight sparsity in neural networks. This method aims to close the accuracy gap between structured and unstructured sparse training, particularly at high sparsity levels. By learning a permutation matrix alongside the weight matrix, SHUFFLESPARSE has shown significant improvements in accuracy for models like ViT-B16 and GPT-2, and notably boosts the zero-shot accuracy of LLaMA-2 7B by 4.6 points. AI

IMPACT This method could enable more efficient deployment of large models by improving structured sparsity techniques.

RANK_REASON The cluster is a research paper detailing a new method for improving neural network sparsity. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Abhishek Tyagi
• arXiv
• GPT-2
• graphics processing unit
• Hugging Face
• LLaMA-2 7B
• SHUFFLESPARSE
• ViT-B16
• WikiText-103

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →