Newton-Schulz
PulseAugur coverage of Newton-Schulz — every cluster mentioning Newton-Schulz across labs, papers, and developer communities, ranked by signal.
4 day(s) with sentiment data
-
MUON-RT optimizer slashes LLM self-distillation memory use, boosts accuracy
Researchers have developed MUON-RT, a novel optimization technique for large language model self-distillation that eliminates the need for separate momentum buffers. By reusing the teacher-student displacement, MUON-RT …
-
New research analyzes last-iterate convergence of Muon for LLM training
A new research paper delves into the theoretical underpinnings of Muon, a method showing strong empirical results in large-scale language model training. The study focuses on the 'last-iterate' convergence of practical …
-
Research paper analyzes Muon optimizer's PolarGD update method
A new research paper published on arXiv analyzes the "PolarGD" update method, a core component of the Muon optimizer. The study, which uses matrix quadratics for analysis, suggests that PolarGD's orthogonalization strat…
-
New research probes Muon optimizer's geometry, performance, and limitations · 4 sources tracked
Four recent arXiv papers explore the optimization algorithm Muon, investigating its theoretical underpinnings and practical performance. The first paper delves into the geometry of Muon, introducing the Random-Radius Is…
-
New Musec Optimizer Enhances LLM Training Stability
Researchers have introduced MomentUm SpEctral Clipping (Musec), a novel optimizer designed to stabilize the training of large language models. Musec addresses instability issues inherent in the Muon optimizer, which oft…
-
New unsupervised method adapts PDE foundation models without ground-truth data
Researchers have developed a new unsupervised finetuning framework for partial differential equation (PDE) foundation models, eliminating the need for ground-truth solutions. This method utilizes a physics-based objecti…
-
New training method improves recurrent memory model performance
A new research paper introduces the "Orthogonalized Read," a training technique designed to improve the performance of recurrent memory models. This method, applied during the read phase of multiplicative LSTMs, acts as…
-
New 'Muse' optimizers explore representation geometry for LLMs
Researchers have introduced "Muse," a novel family of optimizers designed for large language models that explores the geometric properties of parameter representations. Unlike standard Muon-style optimizers, Muse's upda…
-
Turbo-Muon optimizer speeds up AI training with new pre-conditioning technique
Researchers have developed Turbo-Muon, a new pre-conditioning procedure designed to speed up Muon, an optimizer known for its strong performance in large-scale AI training. Turbo-Muon enhances the initialization of the …
-
Matrix Orthogonalization Boosts RNN Memory for Long-Horizon Tasks
Researchers have developed a method to improve the memory capabilities of recurrent neural networks (RNNs) by applying matrix orthogonalization during read operations. This technique, inspired by optimizers used in lang…
-
Muon optimizer needs less orthogonalization than previously thought
Researchers have investigated the optimal level of orthogonalization needed for the Muon optimizer, a technique that enhances neural network training by refining momentum updates. Their study utilized a simplified cubic…
-
Researchers propose novel second-order method for Stiefel manifold optimization
Researchers have developed a novel second-order optimization method for the Stiefel manifold that avoids retractions, offering improved efficiency for high-accuracy requirements. This method combines a tangent component…