Muon optimizer boosts LLM training efficiency over Adam

Researchers have detailed why the Muon optimizer offers improved training efficiency for large language models compared to Adam. Their analysis indicates Muon achieves a greater reduction in loss per step by incurring a smaller penalty related to the curvature of the training landscape. This advantage is primarily due to Muon's lower Normalized Directional Sharpness (NDS), rather than differences in update scale, and is particularly pronounced with imbalanced training data. AI

影响 Explains a key factor in improving LLM training speed and efficiency.

排序理由 Academic paper detailing a novel optimization technique for LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.AI 阅读 →

• Adam
• Large Language Models
• Muon

• 论文
• 其他

AI 生成摘要 · Google Gemini · 来自 1 个来源。 我们如何撰写摘要 →