Muon optimizer shows token efficiency gains on Mamba-2 130M

A new research paper explores the effectiveness of the Muon optimizer, which uses a Newton-Schulz iteration for steepest descent under the spectral norm, on state-space models. The study compares Muon with AdamW on the Mamba-2 130M model, finding that applying Muon to the output projection alone yielded the most significant benefits in token efficiency. This advantage was observed across different corpora and training durations, and it was not explained by conditioning improvements in the input projection. AI

IMPACT This research suggests potential improvements in training efficiency for state-space models, which could influence future model development and optimization strategies.

RANK_REASON The cluster contains a research paper detailing experimental results on an optimizer's performance with a specific model architecture. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• AdamW
• Mamba
• Mamba-2 130M
• Muon
• Newton-Schulz iteration
• Transformer++

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →