AF-Muon optimizer eliminates AdamW for tied-embedding models, saving memory
ByPulseAugur Editorial·[10 sources]·
Researchers have developed AF-Muon, an optimizer that removes the need for AdamW in tied-embedding models, potentially saving around 20% of optimizer-state memory. This new approach maintains Muon's spectral-norm steepest-descent update for hidden weight matrices while employing a support-aware linear minimization for tied vocabulary tables. AF-Muon demonstrates robust improvements in mean validation loss and perplexity across various model architectures and data modalities, including language models, image models, and protein sequences, with minimal overhead.
AI
IMPACT
Optimizers like AF-Muon can reduce memory requirements and improve training efficiency for large models, potentially accelerating research and deployment.
RANK_REASON
Multiple arXiv papers detailing advancements and analyses of the Muon optimizer and its variants.
arXiv:2610.07103v1 Announce Type: cross Abstract: Muon, an optimizer known for its efficiency, has a clear interpretation for matrix-valued updates, but convolutional kernels are stored as four-dimensional tensors. Standard implementations reshape these tensors into matrices, a s…
arXiv:2610.03306v1 Announce Type: cross Abstract: Existing convergence analyses of Muon either assume exact orthogonalization or analyze classical Newton--Schulz polynomials, and guarantee only stationarity, so it is unresolved what Muon's five tuned Newton--Schulz steps preserve…
arXiv:2610.02798v1 Announce Type: new Abstract: The Muon optimizer applies spectral orthogonalization to matrix-valued updates and has shown strong performance in large-scale neural network training, yet the mechanisms of this transformation in feature learning remain poorly unde…
arXiv:2610.01395v1 Announce Type: new Abstract: Muon improves large-scale training by applying a spectral-norm steepest-descent update to matrix parameters, but practical models also contain parameter blocks that do not fit dense-matrix geometry. One important case is the tied vo…
arXiv:2609.39595v1 Announce Type: new Abstract: Practical Muon maintains momentum and performs a small, fixed number of Newton--Schulz iterations separately for each parameter matrix, often with a Nesterov correction. We analyze these layer-wise finite-step updates jointly on a c…
arXiv:2609.39114v1 Announce Type: new Abstract: This paper examines how row-wise renormalization affects Muon, focusing on the gap between NorMuon's worst-case guarantees and its practical performance (Li et al.). Despite its growing adoption and promising performance in large la…
arXiv:2609.37525v1 Announce Type: new Abstract: Physical neural networks and analog in-memory computing could reduce the energy cost of neural network training. Realizing this potential, however, requires optimizers that combine effective learning with physical implementability. …
arXiv cs.AI
TIER_1English(EN)·Arthur C. B. de Oliveira, Dhruv D. Jatkar, Guilherme S. Vicinansa, Eduardo D. Sontag·
arXiv:2609.30546v1 Announce Type: cross Abstract: In this paper, we establish the first asymptotic convergence guarantees for the Muon algorithm through a more accurate proxy for the Newton-Schultz iteration than the typical matrix sign function. We prove that, for appropriate ch…
arXiv:2610.02158v1 Announce Type: cross Abstract: We consider the problem of sampling from Gibbs distributions on matrix spaces whose potential energies are neither convex nor globally gradient-Lipschitz. We introduce a family of non-quadratic kinetic energies that lead to a new …
<h3>Muon: The Orthogonalized Optimizer Explained Through Equations, Intuition, Code and Infographic</h3><h4>From SGD, to Adam’s per-coordinate scaling, to the five Newton–Schulz steps that turn a gradient into an orthogonal update</h4><p>For about a decade, the optimizer was the …
