muon
PulseAugur coverage of muon — every cluster mentioning muon across labs, papers, and developer communities, ranked by signal.
6 day(s) with sentiment data
Muon's neuron death issue may be addressed by new optimizers like Aurora within 3 months
The Tilde Research launch of Aurora specifically targets neuron death in Muon. Given Aurora's public release and demonstrated effectiveness, it's plausible that Muon users will adopt Aurora or similar solutions to mitigate this issue within the next quarter.
Muon's spectral properties are being actively studied in relation to optimizer behavior and mode connectivity
Multiple recent clusters highlight research into Muon's spectral properties and how they interact with optimization dynamics. The connection between optimizers, spectral norms, and mode connectivity suggests ongoing theoretical and empirical work is exploring fundamental aspects of Muon's behavior.
Muon's spectral Wasserstein distances may lead to new theoretical frameworks for deep learning optimization
The introduction of Muon's spectral Wasserstein distances offers a novel approach to understanding deep learning optimization through a continuous-time, mean-field lens. This could inspire further theoretical developments in analyzing and stabilizing training dynamics for wide neural networks.
Muon's performance is being benchmarked against new optimizers like Aurora and OrScale
Multiple recent papers introduce new optimizers (Aurora, OrScale) that explicitly compare themselves against or build upon Muon. This suggests Muon is a relevant baseline in the current research landscape for optimizers.
Muon's theoretical underpinnings may explain its advantage in high-dimensional settings
A new theory suggests that sign-based optimizers like SignSGD can outperform standard SGD in large models due to complexity reduction related to dimensionality. This theory is extended to matrix-based optimizers like Muon, implying Muon may have inherent advantages in high-dimensional parameter spaces.
-
Pion optimizer preserves spectrum for stable LLM training
Researchers have introduced Pion, a novel spectrum-preserving optimizer designed for training large language models. Unlike traditional additive optimizers like Adam, Pion utilizes orthogonal transformations to update w…
-
Tilde Research launches Aurora optimizer to fix neuron death in Muon
Tilde Research has introduced Aurora, a novel optimizer designed to train neural networks more effectively. Aurora addresses a critical issue in the popular Muon optimizer where a significant number of neurons become pe…
-
Muown optimizer improves LLM training by controlling row-norm drift
Researchers have developed Muown, a novel optimization method designed to improve the training of large language models. Muown addresses issues with the Muon optimizer, specifically the upward drift of spectral norms in…
-
New research links optimizers to mode connectivity in neural networks
Researchers have explored the role of optimizers in mode connectivity within neural networks, a concept previously underexplored. Their work demonstrates that solutions generated by a single optimizer, such as AdamW or …
-
Muon framework offers new spectral Wasserstein distances for deep learning
Researchers have introduced a new framework called Muon to stabilize deep-learning optimization using spectral normalizations, particularly for matrix-shaped parameters. This work idealizes the continuous-time, vanishin…
-
Muon optimizer analysis reveals distinct convergence phases vs. SignSGD
Researchers have analyzed stochastic spectral optimizers, including Muon, in a high-dimensional matrix-valued least squares problem. Their analysis reveals that SignSVD, which Muon approximates, performs a square-root p…
-
Aurora optimizer boosts neural network training efficiency
Researchers have introduced Aurora, a new optimizer designed to improve the training of large neural networks, particularly those with rectangular matrices. Aurora addresses issues like neuron death in MLP layers that c…
-
Muon optimizer fails on convex Lipschitz functions, study finds
A new paper challenges the theoretical underpinnings of the Muon optimization algorithm, demonstrating that it does not converge on convex Lipschitz functions. The research suggests that Muon's practical success likely …
-
OrScale optimization method improves neural network training
Researchers have introduced OrScale, a novel optimization technique designed to enhance neural network training. OrScale builds upon the Muon method by incorporating layer-wise trust-ratio scaling, which measures the Fr…
-
Pro-KLShampoo optimizer improves LLM pre-training with spectral structure analysis
Researchers have developed Pro-KLShampoo, an optimization technique that combines gradient preconditioning with orthogonalization for more efficient LLM pre-training. This method leverages the observed spike-and-flat ei…
-
New research links optimizer choice to reduced forgetting in LLM finetuning
Researchers have explored the impact of optimizer consistency during the fine-tuning of large language models. One study suggests that using the same optimizer for both pre-training and fine-tuning leads to less knowled…
-
New LMO-IGT method accelerates optimization with implicit gradient transport
Researchers have introduced LMO-IGT, a novel class of stochastic optimization methods designed to accelerate convergence in machine learning. This approach leverages implicit gradient transport (IGT) to achieve faster r…
-
SignSGD and Muon optimizers' performance gains theoretically explained
Researchers have theoretically analyzed why sign-based optimization algorithms like SignSGD and Muon can outperform standard SGD in training large models. A new study suggests that SignSGD's advantage stems from its eff…
-
New Polar Express method accelerates matrix decomposition for deep learning
Researchers have developed a new GPU-friendly algorithm called Polar Express for computing matrix decompositions, which is crucial for the Muon optimizer used in training deep neural networks. This method optimizes for …
-
Nora optimizer achieves efficiency, stability, and speed for large-scale LLM training
Researchers have introduced Nora, a novel matrix-based optimizer designed for efficient and stable training of large language models. Nora aims to unify efficiency, stability, and speed, addressing limitations of existi…
-
New theory unifies adaptive optimization methods for nonconvex machine learning
Researchers have developed a unified framework to analyze first-order optimization algorithms used in nonconvex machine learning. This framework encompasses popular methods like AdaGrad, AdaNorm, and variants of Shampoo…
-
Spectral optimizers like Muon show sharp capacity scaling in associative memory tasks
A new paper analyzes the performance of spectral optimizers, like Muon, in training large language models by examining their effectiveness in learning associative memory. The research demonstrates that Muon significantl…
-
DeepSeek V4 models offer high performance with reduced inference costs and NPU support
DeepSeek has released its V4 family of open-weight large language models, featuring a 1.6 trillion parameter model and a smaller 284 billion parameter Flash MoE model. These new models claim to rival top proprietary LLM…