New Muon Optimizer Variants Enhance LLM Training Efficiency and Performance
ByPulseAugur Editorial·[17 sources]·
Multiple research papers explore advancements and applications of the Muon optimizer for training large language models and other deep learning architectures. MONA introduces Nesterov acceleration to Muon for improved convergence and downstream task performance, achieving state-of-the-art results on large models. MuCon investigates clipping Muon updates to approximate its behavior without full singular value decomposition. Another study examines Muon's effectiveness in adversarial training, showing it can be competitive with or outperform standard optimizers like SGD and AdamW across various threat models and architectures. Further research introduces DynMuon for dynamic spectral shaping, HTMuon for heavy-tailed spectral correction, and AMUSE for stable gradient evaluation, all aiming to enhance Muon's performance and training efficiency.
AI
IMPACT
These advancements in optimizers like MONA, DynMuon, HTMuon, and AMUSE promise to accelerate large language model training and improve performance on various downstream tasks.
RANK_REASON
Multiple arXiv papers detailing new theoretical and empirical studies of the Muon optimizer and its variants for large language model training.
arXiv:2605.26842v1 Announce Type: cross Abstract: The Muon optimizer has recently offered a promising alternative to AdamW for large language model training, leveraging matrix orthogonalization to produce geometry-aware updates. However, like all first-order methods, Muon can bec…
arXiv:2605.26459v1 Announce Type: new Abstract: Muon-style optimizers take a matrix-valued momentum or preconditioned update $B = U \operatorname{diag}(\sigma_1,\ldots,\sigma_r) V^\top$ and replace it with its canonical partial polar factor $\operatorname{Pol}(B) = U V^\top$. Thi…
arXiv:2605.26929v1 Announce Type: new Abstract: Adversarial training (AT) remains one of the most reliable empirical defenses against adversarial attacks. Its robustness critically depends on how the underlying min-max objective is optimized. In practice, Stochastic Gradient Desc…
Adversarial training (AT) remains one of the most reliable empirical defenses against adversarial attacks. Its robustness critically depends on how the underlying min-max objective is optimized. In practice, Stochastic Gradient Descent (SGD) optimizer remains the default optimiza…
The Muon optimizer has recently offered a promising alternative to AdamW for large language model training, leveraging matrix orthogonalization to produce geometry-aware updates. However, like all first-order methods, Muon can become trapped in sharp local minima. In this work, w…
arXiv:2602.05725v2 Announce Type: replace Abstract: Muon updates matrix parameters via the matrix sign of the gradient and has shown strong empirical gains, yet its dynamics and scaling behavior remain unclear in theory. We study Muon in a linear associative memory model with sof…
arXiv cs.LG
TIER_1English(EN)·Ben S. Southworth, Shuai Jiang, Daniel McBride, Eric C. Cyr, Stephen Thomas·
arXiv:2605.24770v1 Announce Type: new Abstract: Muon is a recently developed matrix-aware optimizer that has shown strong results in transformer training, but its behavior in vision transformers (ViTs) is not yet well understood. We study Muon for ViT training, largely on ImageNe…
arXiv:2605.17109v2 Announce Type: replace-cross Abstract: In recent years, Muon has emerged as the dominant method for training large language models, and transformers more broadly. The essential difference, when compared to standard gradient descent methods, is to replace the us…
arXiv:2603.10067v2 Announce Type: replace-cross Abstract: Muon has recently shown promising results in LLM training. In this work, we study how to further improve Muon. We argue that Muon's orthogonalized update rule suppresses the emergence of heavy-tailed weight spectra and ove…
arXiv:2605.22432v1 Announce Type: new Abstract: Modern deep learning commonly relies on AdamW with prescribed learning rate schedules, but recent works challenge both components: Schedule-Free optimization removes explicit schedules via iterate averaging, and Muon improves the up…
Modern optimizers, like Muon, impose matrix-wise geometry constraints on their updates. These matrix-wise constraints can be unified under Linear Minimization Oracle (LMO) theory. However, all current methods impose fixed LMO geometries for the update rules, chosen by-design or e…
Muon is a matrix-aware optimizer that leverages Newton-Schulz (NS) iterations to enforce spectral gradient orthogonalization by driving all singular values of the momentum matrix toward 1. While this uniform spectral whitening enhances exploration and outperforms AdamW in LLM pre…
Muon's spectral whitening approach in LLM pretraining is replaced by Pion, which uses a high-pass NS iteration to stabilize training in low-rank and low-SNR regimes while maintaining computational efficiency and supporting per-head updates.
arXiv stat.ML
TIER_1English(EN)·Aratrika Mustafi, Soumya Mukherjee, Bharath K. Sriperumbudur·
arXiv:2605.23871v1 Announce Type: new Abstract: We develop a gradient flow on the space of probability measures defined on matrix-valued parameters induced by regularized Muon, an analytically smoothed version of the idealized Muon optimizer. The key observation is that the regul…
arXiv stat.ML
TIER_1English(EN)·Bharath K. Sriperumbudur·
We develop a gradient flow on the space of probability measures defined on matrix-valued parameters induced by regularized Muon, an analytically smoothed version of the idealized Muon optimizer. The key observation is that the regularized orthogonalization map is the gradient of …
arXiv:2605.19619v1 Announce Type: cross Abstract: Matrix-structured parameters frequently appear in many artificial intelligence models such as large language models. More recently, an efficient Muon optimizer is designed for matrix parameters of large-scale models, and shows mar…
Matrix-structured parameters frequently appear in many artificial intelligence models such as large language models. More recently, an efficient Muon optimizer is designed for matrix parameters of large-scale models, and shows markedly faster convergence than the vector-wise algo…
