Polyak
PulseAugur coverage of Polyak — every cluster mentioning Polyak across labs, papers, and developer communities, ranked by signal.
-
Researchers detail modified loss in momentum gradient descent
Researchers have published a fine-grained analysis of Polyak's heavy-ball momentum gradient descent algorithm. The study proves that under certain conditions, the algorithm behaves like plain gradient descent with a mod…
-
AF-Muon optimizer eliminates AdamW for tied-embedding models, saving memory
Researchers have developed AF-Muon, an optimizer that removes the need for AdamW in tied-embedding models, potentially saving around 20% of optimizer-state memory. This new approach maintains Muon's spectral-norm steepe…
-
New research details momentum's role in large-batch AI training
A new research paper explores how momentum impacts large-batch training in machine learning, using power-law kernel regression as a model. The study defines a critical learning rate to characterize risk stability and de…