MuonQ framework enables 4-bit quantization for LLM training, cutting memory use

Researchers have developed MuonQ, a novel framework designed to enable low-bit quantization of the Muon optimizer for training large language models. MuonQ addresses the sensitivity of the Muon optimizer to quantization errors by employing pre-quantization normalization, structural decomposition to protect directional information, and $\mu$-law companding quantization for better resolution allocation. Experiments with GPT-style and LLaMA-style models show that MuonQ at 4-bit precision can achieve training loss and downstream accuracy comparable to full-precision Muon, while significantly reducing optimizer-state memory by up to 7.3x. AI

IMPACT Enables significant memory reduction for LLM training, potentially making larger models more accessible.

RANK_REASON The item is an academic paper detailing a new method for optimizing LLM training. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Adam
• generative pre-trained transformer
• llama
• Muon
• MuonQ
• Yupeng Su

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →