A new optimization algorithm named Muon has demonstrated an iteration complexity of $\mathcal{O}(\varepsilon^{-3})$ for finding an $\varepsilon$-stationary point in smooth, non-convex finite-sum problems when using shuffling. This complexity matches that of stochastic gradient descent under shuffling and improves upon Muon's previous $\mathcal{O}(\varepsilon^{-4})$ complexity with sampling with replacement. The research also highlights a failure case for Muon with a fixed momentum coefficient $\beta$, showing it can fail to converge to stationarity for certain problem configurations, suggesting practical tuning rules for increasing $\beta$ and reducing stepsize when training loss plateaus. AI
IMPACT This research could lead to more efficient training of large language models by improving optimization algorithms.
RANK_REASON Academic paper detailing a new optimization algorithm and its theoretical complexity. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →