AdaGrad
PulseAugur coverage of AdaGrad — every cluster mentioning AdaGrad across labs, papers, and developer communities, ranked by signal.
1 day(s) with sentiment data
-
New continuous-time models for AdaGrad, RMSProp, and Adam
Researchers have developed a continuous-time framework to model popular optimization algorithms like AdaGrad, RMSProp, and Adam. By representing these algorithms as integro-differential equations, the study provides a n…
-
New OptMuon method enhances stochastic optimization with adaptive momentum
Researchers have introduced OptMuon, a novel adaptive momentum orthogonalization method for stochastic nonconvex optimization that calibrates update magnitudes from observed trajectories. This approach combines Muon-sty…
-
Muon optimizer fails on convex Lipschitz functions, study finds
A new paper challenges the theoretical underpinnings of the Muon optimization algorithm, demonstrating that it does not converge on convex Lipschitz functions. The research suggests that Muon's practical success likely …
-
LLM Study Diary #3: PyTorch tensors, float types, and training infrastructure
This LLM study diary entry focuses on PyTorch fundamentals for training large language models. It details tensor basics, exploring various floating-point data types like FP32, BF16, and FP8 for efficiency and stability.…
-
FG^2-GDN enhances long-context understanding with adaptive learning rates
Researchers have introduced FG$^2$-GDN, a novel approach to enhance long-context understanding in neural networks. This method improves upon existing Gated Delta Networks by replacing a scalar learning rate with a chann…
-
New theory unifies adaptive optimization methods for nonconvex machine learning
Researchers have developed a unified framework to analyze first-order optimization algorithms used in nonconvex machine learning. This framework encompasses popular methods like AdaGrad, AdaNorm, and variants of Shampoo…