Nesterov momentum
PulseAugur coverage of Nesterov momentum — every cluster mentioning Nesterov momentum across labs, papers, and developer communities, ranked by signal.
1 day(s) with sentiment data
-
Nesterov momentum stability analyzed for large neural networks
Researchers have analyzed the stability of Nesterov momentum when applied to stochastic pseudo-gradients in large neural networks trained across multiple workers. They derived a mean-square stability criterion for rando…
-
AF-Muon optimizer eliminates AdamW for tied-embedding models, saving memory
Researchers have developed AF-Muon, an optimizer that removes the need for AdamW in tied-embedding models, potentially saving around 20% of optimizer-state memory. This new approach maintains Muon's spectral-norm steepe…
-
New inference method for Newton methods accelerates convergence
Researchers have developed a novel online sketched Newton method that uses a generalized accelerated sketch-and-project solver (GAS) to approximate Newton directions, addressing the computational bottleneck of tradition…
-
New NoLoCo method slashes LLM training communication costs
Researchers have developed a novel optimization method called NoLoCo, designed to significantly reduce communication overhead during the training of large language models. Unlike existing low-communication methods that …
-
New framework accelerates weighted low-rank matrix approximation methods
This paper introduces Weighted Low-Rank Matrix Approximation (WLRMA) as a generalization of classical low-rank approximation and matrix completion. It presents a unified framework for developing efficient optimization m…
-
Model merging techniques enhance distributed learning in new IsoLoCo approach
Researchers have explored the use of model merging techniques to improve aggregation in distributed learning methods like DiLoCo. By drawing an analogy between pseudo-gradient aggregation in local SGD/DiLoCo and task ar…
-
Research paper analyzes compute efficiency and runtime tradeoffs for momentum methods
A new research paper explores the tradeoffs between serial runtime and compute efficiency for stochastic momentum methods like Heavy Ball (HB) and Accelerated SGD (ASGD). The study proves finite-dimensional lower bounds…
-
New rod flow model tracks Adam optimizer at edge of stability
Researchers have developed a new "rod flow" model to better understand how adaptive gradient optimization methods, like Adam, operate at the edge of stability. This model extends previous work on gradient descent to inc…
-
An Integrated Framework for Explainable, Fair, and Observable Hospital Readmission Prediction: Development and Validation on MIMIC-IV
Researchers have developed a new gradient-regularized Newton scheme to ensure global convergence for Gradient Boosting Decision Trees (GBDTs), a technique widely used in tabular machine learning. This method introduces …