Researchers have introduced KronQ, a novel post-training quantization (PTQ) framework designed to compress large language models (LLMs) more effectively. Unlike previous methods that rely solely on activation statistics, KronQ incorporates gradient covariance into its quantization objectives. This approach, utilizing a Kronecker-factored Hessian approximation, leads to improved weight magnitude variance and a new metric for mixed-precision allocation. In experiments with LLaMA-3-70B, KronQ achieved a perplexity of 7.93 for 2-bit weight-only quantization, significantly outperforming existing methods like GPTQ and GPTAQ which resulted in degenerate quantizations. AI
IMPACT This new quantization technique could lead to more efficient deployment and reduced computational costs for large language models.
RANK_REASON The item describes a new research paper detailing a novel framework for LLM quantization. [lever_c_demoted from research: ic=1 ai=1.0]
Read on Hugging Face Daily Papers →
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
