AdamW
PulseAugur coverage of AdamW — every cluster mentioning AdamW across labs, papers, and developer communities, ranked by signal.
17 day(s) with sentiment data
-
Muon-trained transformers exhibit post-grokking collapse, losing generalization
A new research paper explores a phenomenon termed "post-grokking collapse" in transformers trained with the Muon optimizer. While Muon initially demonstrates faster learning on modular addition tasks, the models eventua…
-
New SG-TULA algorithm offers improved sampling for complex AI models
Researchers have developed the Subgradient Tamed Unadjusted Langevin Algorithm (SG-TULA), a novel method for sampling from complex distributions that are non-smooth, non-convex, and have superlinear gradient growth. Thi…
-
New research probes MUON optimizer's convergence and proposes MALT extension
Two new research papers explore the MUON optimization algorithm, a method used in training large language models. The first paper introduces MALT, an extension of MUON that incorporates lightweight diagonal precondition…
-
New MESH optimizer boosts MoE training efficiency, cuts memory use
Researchers have developed MESH, a novel optimization technique designed to improve the efficiency of training Mixture-of-Experts (MoE) models. Traditional memory-efficient optimizers like Sinkhorn struggle with MoE arc…
-
Muon optimizer shows token efficiency gains on Mamba-2 130M
A new research paper explores the effectiveness of the Muon optimizer, which uses a Newton-Schulz iteration for steepest descent under the spectral norm, on state-space models. The study compares Muon with AdamW on the …
-
New AOS-R system optimizes deep learning training by switching optimizers
Researchers have developed AOS-R, a novel adaptive optimizer switching system designed to improve deep network training efficiency and generalization. This system monitors six online gradient-space signals to dynamicall…
-
New nGPT training method halves token requirements for MoE models
Researchers have developed a new training methodology called normalized Transformer (nGPT) that constrains model parameters and activations to a unit hypersphere for improved representation learning. This recipe, detail…
-
New research reveals high-entropy models resist catastrophic forgetting better than AdamW
A new research paper titled "The Grokked Illusion: True Equilibrium Mitigates Catastrophic Forgetting" explores the robustness of neural network representations beyond standard generalization metrics. The study found th…
-
New CAP-TTA framework enhances LLM debiasing for narrative generation
Researchers have developed CAP-TTA, a novel test-time adaptation framework designed to improve the debiasing of large language models (LLMs) when encountering out-of-distribution, high-bias prompts. This framework utili…
-
AI model accurately grades acne severity using transfer learning
Researchers have developed a four-class acne severity classifier using transfer learning with an EfficientNet-B0 model, fine-tuned on the ACNE04 dataset. The model achieved 93.5% accuracy and 94.4% macro-F1 on a test se…
-
Muon optimizer shows promise in theoretical and practical neural network training
Two new research papers explore the Muon optimizer, an approach designed to better handle matrix-structured parameters in neural networks. The first paper introduces a matrix-aware geometry for Sharpness-Aware Minimizat…
-
New Transformer Optimization Insight: Hidden Boundary Motion Identified
Researchers have identified a phenomenon called "hidden boundary motion" in Transformer optimization, where weight and bias updates are functionally indistinguishable for affine layers with non-zero mean inputs. This mo…
-
NVIDIA research suggests AdamW optimizer has scale ceiling
NVIDIA researchers have published new findings suggesting that the AdamW optimizer may have a scale ceiling. Their work indicates that at batch sizes up to 100 million tokens for next-token prediction, optimizers like S…
-
New optimizer CvAdamW accelerates neural network grokking
Researchers have introduced CvAdamW, a novel variant of the AdamW optimizer designed to accelerate the "grokking" phenomenon in neural networks. Grokking, where a model generalizes after memorizing training data, is oft…
-
Muon optimizer's grokking speed attributed to orthogonalization, study finds
A new research paper published on arXiv details the active ingredient behind the Muon optimizer's faster grokking threshold in modular arithmetic compared to AdamW. The study isolates orthogonalization, specifically the…
-
New RELTA-SGLD scheme stabilizes stochastic-gradient learning
Researchers have introduced RELTA-SGLD, a new taming scheme designed to stabilize stochastic-gradient updates in nonconvex settings. This method aims to reduce unnecessary suppression of learning drift by employing a th…
-
New research explores adaptive rank allocation for efficient LLM fine-tuning
Two new research papers introduce advanced methods for parameter-efficient fine-tuning (PEFT) of large language models. The first paper proposes LAARA, a framework that dynamically allocates adapter ranks to different t…
-
New RELTA-SGLD method stabilizes SGLD for nonconvex learning
Researchers have developed RELTA-SGLD, a novel taming scheme designed to stabilize stochastic-gradient Langevin dynamics (SGLD) with superlinear growth. This method uses a threshold to activate taming and a relative-gro…
-
New ISO framework optimizes RLVR for language models with fewer training steps
Researchers have introduced Isospectral Optimization (ISO), a new framework designed to improve the efficiency of reinforcement learning with verifiable rewards (RLVR) in language models. ISO leverages the concept of sp…
-
Research paper analyzes loss landscape of two-layer ReLU networks
A new research paper published on arXiv explores the loss landscape of two-layer ReLU networks, focusing on the impact of width-dependent hyperparameters and L2 regularization. The study derives conditions under which g…