Muon optimizer boosts LLM training efficiency over Adam

Researchers have detailed why the Muon optimizer offers improved training efficiency for large language models compared to Adam. Their analysis indicates Muon achieves a greater reduction in loss per step by incurring a smaller penalty related to the curvature of the training landscape. This advantage is primarily due to Muon's lower Normalized Directional Sharpness (NDS), rather than differences in update scale, and is particularly pronounced with imbalanced training data. AI

IMPACT Explains a key factor in improving LLM training speed and efficiency.

RANK_REASON Academic paper detailing a novel optimization technique for LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Adam
• Large Language Models
• Muon

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →