New method slashes LLM quantization bit-width with spectral rotations

Researchers have developed a novel method called BBT-spectral for quantizing large language models (LLMs) to extremely low bit-widths, specifically W2A16 (2-bit weights, 16-bit activations). This technique utilizes influence-inspired spectral rotations and a reconstruction-error quantizer to significantly reduce perplexity, outperforming vanilla auto-round quantization by 15-58% on various model sizes. The method has been extended to address specific architectural challenges in models like Qwen3 and Qwen2.5, demonstrating its adaptability and effectiveness across different LLM families. AI

影响 This research could enable more efficient deployment of LLMs on resource-constrained hardware by significantly reducing their memory footprint.

排序理由 The cluster contains an academic paper detailing a new method for LLM quantization. [lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.AI 阅读 →

• AQLM
• BBT-spectral
• ButterflyQuant
• Intel
• LLM
• OmniQuant
• OpenVINO
• QuaRot
• QuIP-sharp
• Qwen2.5
• Qwen3
• W2A16

• 论文
• 模型发布

AI 生成摘要 · Google Gemini · 来自 1 个来源。 我们如何撰写摘要 →