New Musec Optimizer Enhances LLM Training Stability

Researchers have introduced MomentUm SpEctral Clipping (Musec), a novel optimizer designed to stabilize the training of large language models. Musec addresses instability issues inherent in the Muon optimizer, which often outperforms Adam and AdamW but can suffer from loss spikes and unbounded weight growth. Unlike previous methods that require architectural modifications, Musec implements an optimizer-level spectral clipping mechanism that preserves the momentum matrix's spectral structure. The paper also details an efficient implementation called Soft Musec, which provides theoretical convergence guarantees for non-convex, non-smooth stochastic optimization and demonstrates empirical stability improvements across various learning rates and model sizes. AI

IMPACT Introduces a more stable and efficient method for training large language models, potentially improving performance and reducing divergence issues.

RANK_REASON The cluster contains a research paper detailing a new method for optimizing large language model training. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Adam
• AdamW
• Hugging Face
• large language model
• MomentUm SpEctral Clipping
• Muon
• Newton-Schulz
• Soft Musec

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →