AdamW
PulseAugur coverage of AdamW — every cluster mentioning AdamW across labs, papers, and developer communities, ranked by signal.
11 day(s) with sentiment data
-
7 PhD students train 7B LLM from scratch using hundreds of AI agents
Seven doctoral students from Beijing Zhongguancun Academy successfully trained a 7B large language model, ZGCM-1, from scratch in just three months. They achieved this by leveraging a team of hundreds of AI agents to ha…
-
New theory explains heavy-tail emergence in neural optimizer dynamics
Researchers have developed a new method to understand how heavy-tailed spectral densities emerge in neural network weight matrices, which are indicators of implicit self-regularization. They formulated this emergence as…
-
New ISO-LoRA optimizer boosts parameter-efficient adaptation for LLMs
Researchers have introduced ISO-LoRA, a novel optimization technique designed to enhance the efficiency of Low-Rank Adaptation (LoRA) for large language models. Unlike traditional LoRA methods that focus solely on the r…
-
New methods tackle memory peaks for long-context MoE training
Researchers have developed four novel techniques to address memory limitations in training Mixture-of-Experts (MoE) models with long contexts. These methods, PipelinedLLEP, Ring-DTP, Selective Checkpoint Offload (SCO), …
-
New Musec Optimizer Enhances LLM Training Stability
Researchers have introduced MomentUm SpEctral Clipping (Musec), a novel optimizer designed to stabilize the training of large language models. Musec addresses instability issues inherent in the Muon optimizer, which oft…
-
New research optimizes LoRA fine-tuning for AI models · 2 papers
Two new research papers explore optimizations for Low-Rank Adaptation (LoRA) fine-tuning. The first paper investigates the trade-offs between LoRA rank, model quality, and computational cost, finding that moderate ranks…
-
AI training fundamentals: Gradient descent and AdamW explained
Two articles from Towards AI delve into the fundamental concepts of machine learning training. The first article explains the limitations of gradients in training large models, highlighting the necessity of optimizers l…
-
New research shows optimizer performance shifts with training horizon
A new research paper explores how different optimizers perform as training horizons and parameter counts increase. The study found that the optimal hyperparameters and relative performance of optimizers like Muon, SOAP,…
-
New CAHR-Net model offers compact and interpretable magnetic core loss modeling
Researchers have developed CAHR-Net, a novel network designed for modeling magnetic core loss. This model uniquely integrates operating conditions like frequency, temperature, and waveform shape directly into the hyster…
-
SGD Dynamics Modeled as Percolation Process, Extending to Adam and AdamW
Researchers have modeled the dynamics of Stochastic Gradient Descent (SGD) as a percolation process, revealing how architectural symmetries cause subnetworks to merge in discrete blocks. These transitions result in vari…
-
Study reveals optimal fine-tuning strategies for Qwen3 and Llama models
A new study explores the optimal hyperparameters for supervised fine-tuning (SFT) of large language models, investigating factors like learning rate, batch size, and optimizer choice. The research systematically tested …
-
New research details RMSprop optimizer convergence rates
A new research paper published on arXiv details convergence rates for the RMSprop optimizer, a popular method for training AI systems. The study provides a theoretical solution to the open problem of bounding error cons…
-
OrScale optimization method enhances neural network training
Researchers have introduced OrScale, a novel optimization method designed to improve the training of large neural networks. OrScale addresses the direction and magnitude of updates by adapting the trust-ratio principle …
-
New research explores advanced fine-tuning techniques for LLMs · 3 sources tracked
Three new research papers explore advanced techniques for supervised fine-tuning (SFT) of large language models. The first paper investigates optimal hyperparameters like learning rate and batch size across different mo…
-
Survey reveals neural network optimizers have expanded beyond Adam variants
A recent survey of neural network optimization techniques from 2025-2026 reveals a significant expansion beyond simple Adam variants. The field now explores optimizers that operate on matrices and layers, adapt to dynam…
-
New LLM Pretraining Method Accelerates Training on Flat Directions
Researchers have developed a new method called Curvature-Conditioned Multiscale Momentum with Sphere Constraints to improve the efficiency of Large Language Model (LLM) pretraining. This technique addresses challenges p…
-
Muon optimizer tackles task interference in continual learning and model merging
Researchers have introduced a new perspective on continual learning and model merging, framing both as instances of "task interference." This interference, quantified by a layer-wise Frobenius inner product, is influenc…
-
New Adaptive Log-Space Quantization Boosts LLM Optimizer Memory Efficiency
Researchers have developed a new quantization method called Adaptive Log-Space (AL) to improve the memory efficiency of optimizers used in training large language models. This method adapts the quantization range per bl…
-
New algorithm QEF-GT-AdamW enhances decentralized learning for wireless IoT
A new algorithm called QEF-GT-AdamW has been proposed for decentralized learning in wireless IoT networks. This method aims to improve reliability and reduce communication overhead in environments with heterogeneous dat…
-
Optimizer state transport impacts short-horizon AI training decisions
Researchers have investigated how adaptive optimizers, such as AdamW, use historical gradient information to influence future training decisions. Their study focused on the impact of delayed optimizer-state transport on…