muon
PulseAugur coverage of muon — every cluster mentioning muon across labs, papers, and developer communities, ranked by signal.
17 day(s) with sentiment data
Aurora optimizer may outperform Muown in addressing Muon's neuron death
Tilde Research's Aurora optimizer is specifically designed to fix 'neuron death' in Muon, a problem not explicitly addressed by Muown. While Muown improves spectral norm drift, Aurora's targeted approach to neuron inactivity could lead to more comprehensive performance gains, especially in scenarios where neuron death is a primary bottleneck.
Muon optimizer's spectral norm drift is a key area for improvement
Multiple recent papers (Muown, Pion, and the general mode connectivity research) highlight issues related to spectral norms and Muon. Muown explicitly addresses 'upward drift of spectral norms', while Pion aims to 'preserve spectrum'. This suggests that managing spectral properties is a critical challenge for Muon's stability and performance.
Spectrum preservation is a common theme in new optimizer research
The introduction of Pion, which 'preserves spectrum', and Muown, which addresses 'spectral norm drift', indicates a broader trend in optimizer development. This focus on maintaining spectral properties suggests that current optimizers, including Muon, may suffer from spectral instability that hinders training.
Muon's spectral properties are being actively studied in relation to optimizer behavior and mode connectivity
Multiple recent clusters highlight research into Muon's spectral properties and how they interact with optimization dynamics. The connection between optimizers, spectral norms, and mode connectivity suggests ongoing theoretical and empirical work is exploring fundamental aspects of Muon's behavior.
Muon's neuron death issue may be addressed by new optimizers like Aurora within 3 months
The Tilde Research launch of Aurora specifically targets neuron death in Muon. Given Aurora's public release and demonstrated effectiveness, it's plausible that Muon users will adopt Aurora or similar solutions to mitigate this issue within the next quarter.
-
New GO-MUON method refines spectral geometry for optimization
Researchers have introduced GO-MUON, a novel optimization method that enhances the accuracy of muon's polar update by employing a matched data-dependent geometry. This approach precisely solves weighted spectral oracles…
-
Muon-trained transformers exhibit post-grokking collapse, losing generalization
A new research paper explores a phenomenon termed "post-grokking collapse" in transformers trained with the Muon optimizer. While Muon initially demonstrates faster learning on modular addition tasks, the models eventua…
-
New Skewon algorithm offers exact closed-form optimization on Stiefel manifold
Researchers have developed Skewon, a new optimization algorithm for problems involving matrices with orthonormal columns, a common structure in machine learning. This algorithm provides an exact closed-form solution for…
-
New SG-TULA algorithm offers improved sampling for complex AI models
Researchers have developed the Subgradient Tamed Unadjusted Langevin Algorithm (SG-TULA), a novel method for sampling from complex distributions that are non-smooth, non-convex, and have superlinear gradient growth. Thi…
-
Kimi K3 leverages 896 experts and hybrid attention for efficient scaling
Kimi K3, a 2.8 trillion parameter model, employs a novel approach to manage its massive scale by activating only 16 out of 896 routing experts per token. This strategy, detailed by researcher Su Jianlin, aims to control…
-
Adam optimizer differs from gradient descent in factored models
A new research paper explores the differing behaviors of optimization algorithms like Adam and gradient descent when applied to factored models. The study reveals that while gradient descent is implicitly biased towards…
-
New research probes MUON optimizer's convergence and proposes MALT extension
Two new research papers explore the MUON optimization algorithm, a method used in training large language models. The first paper introduces MALT, an extension of MUON that incorporates lightweight diagonal precondition…
-
Muon optimizer shows token efficiency gains on Mamba-2 130M
A new research paper explores the effectiveness of the Muon optimizer, which uses a Newton-Schulz iteration for steepest descent under the spectral norm, on state-space models. The study compares Muon with AdamW on the …
-
New method Joint Affine Spectral Shaping improves neural network optimization
Researchers have developed a new method called Joint Affine Spectral Shaping (JASS) that improves upon existing spectral optimizers for neural networks. Unlike previous methods that treated weight and bias updates separ…
-
Gradient Descent Favors Low-Rank Solutions Over Adam in Factored Models
A new paper explores the differing behaviors of optimization algorithms, specifically gradient descent and Adam, when applied to factored matrix models. The research indicates that gradient descent inherently favors low…
-
New SignMuon method compresses AI model updates to one bit per parameter
Researchers have developed SignMuon, a method for compressing model updates to a single bit per parameter, significantly reducing communication overhead. While SignMuon outperforms SignSGD in practice, it can still dive…
-
Moonshot AI open-sources Kimi K3, valuation hits $31.5B · 2 sources tracked
Moonshot AI has fully open-sourced its Kimi K3 model, a 2.8T parameter model, and revealed the 401 contributors behind it. This release coincides with the company's valuation soaring to $31.5 billion, with each employee…
-
Muon optimizer shows promise in theoretical and practical neural network training
Two new research papers explore the Muon optimizer, an approach designed to better handle matrix-structured parameters in neural networks. The first paper introduces a matrix-aware geometry for Sharpness-Aware Minimizat…
-
New scaled weight decay method accelerates neural network training
Researchers have introduced a novel method for scaling weight decay, inspired by the Robbins-Monro conditions, to improve neural network training. This technique adjusts weight decay based on the fraction of the peak le…
-
New research explores quantum and Frank-Wolfe methods for heavy-tailed noise optimization
Two new research papers explore optimization techniques under heavy-tailed noise conditions. The first paper, "Quantum Speedups for Stochastic Optimization with Heavy-Tailed Noise," proposes novel quantum mean estimator…
-
NVIDIA research suggests AdamW optimizer has scale ceiling
NVIDIA researchers have published new findings suggesting that the AdamW optimizer may have a scale ceiling. Their work indicates that at batch sizes up to 100 million tokens for next-token prediction, optimizers like S…
-
MuonQ framework enables 4-bit quantization for LLM training, cutting memory use
Researchers have developed MuonQ, a novel framework designed to enable low-bit quantization of the Muon optimizer for training large language models. MuonQ addresses the sensitivity of the Muon optimizer to quantization…
-
Muon optimizer's grokking speed attributed to orthogonalization, study finds
A new research paper published on arXiv details the active ingredient behind the Muon optimizer's faster grokking threshold in modular arithmetic compared to AdamW. The study isolates orthogonalization, specifically the…
-
New spectral cap method enhances LLM training by controlling weight matrix geometry
Researchers have proposed a new method called an "Isotropy-Preserving Spectral Cap" to improve the training of large language models (LLMs). This technique aims to control the internal geometry of weight matrices during…
-
New ISO framework optimizes RLVR for language models with fewer training steps
Researchers have introduced Isospectral Optimization (ISO), a new framework designed to improve the efficiency of reinforcement learning with verifiable rewards (RLVR) in language models. ISO leverages the concept of sp…