NVIDIA researchers have published new findings suggesting that the AdamW optimizer may have a scale ceiling. Their work indicates that at batch sizes up to 100 million tokens for next-token prediction, optimizers like SOAP and Muon maintain training stability and quality, whereas AdamW degrades. The researchers addressed instabilities in SOAP at large batch sizes through orthogonalization and preconditioning, demonstrating that both Muon and SOAP consistently outperform AdamW on multi-billion-parameter models trained over trillions of tokens. This research also introduces a layer-wise distributed optimizer compatible with Megatron-LM. AI
IMPACT Suggests potential limitations of a widely used optimizer, potentially driving adoption of alternatives for large-scale model training.
RANK_REASON New research paper detailing findings on AI model training optimizers. [lever_c_demoted from research: ic=1 ai=1.0]
Read on X — Omar Sanseviero (HF research) →
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
