Researchers have developed two new methods, Layer-wise Error Modeling Search (LEMS) and KFAC-SVD, to improve the compression of large language models using singular value decomposition (SVD). LEMS enhances rank allocation by considering layer-wise error surrogates and propagation bias, while KFAC-SVD improves decomposition quality by using token-wise statistics to mitigate rank collapse. These techniques have demonstrated significant zero-shot accuracy gains of up to 4.8 percentage points and an average perplexity improvement of 15% across various models, including Mistral, Qwen3, and Llama 3, and scale to models up to 70 billion parameters. AI
IMPACT These techniques could lead to more efficient deployment of large language models, reducing computational costs and enabling wider accessibility.
RANK_REASON The cluster describes a research paper detailing new methods for LLM compression. [lever_c_demoted from research: ic=1 ai=1.0]
Read on Hugging Face Daily Papers →
- International Conference on Machine Learning
- KFAC-SVD
- Layer-wise Error Modeling Search
- LEMS
- Llama 3
- Mistral AI
- MoritzMo123/kfac-svd_lems_llama-3-8b_0.8
- Qwen3
- singular value decomposition
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →