SGD
PulseAugur coverage of SGD — every cluster mentioning SGD across labs, papers, and developer communities, ranked by signal.
- instance of ScienceCast 90%
- instance of DagsHub 90%
- instance of stochastic gradient descent 90%
- used by Neural tangent kernel 80%
- competes with AdamW 70%
- used by AdamW 70%
- authored by alphaXiv 70%
- authored Gotit.pub 70%
- authored ScienceCast 70%
- instance of AdamW 70%
- used by Gotit.pub 70%
- used by alphaXiv 70%
15 day(s) with sentiment data
-
New optimization theory unifies DNN convexity and smoothness
Researchers have introduced a novel optimization framework for deep neural networks (DNNs) that generalizes classical convexity and smoothness concepts. This new framework, termed $\mathcal{H}(\psi)$-convexity and $\mat…
-
AI training efficiency: Gradient optimization methods benchmarked
A new research paper benchmarks five gradient optimizers and three memory strategies for AI training on constrained hardware. The study found that gradient accumulation is the most effective strategy for reducing traini…
-
New framework enables online statistical inference for complex AI algorithms
Researchers have developed a novel online statistical inference framework for nonlinear stochastic approximation algorithms that utilize Markovian data. This framework establishes a functional central limit theorem for …
-
New research probes MUON optimizer's convergence and proposes MALT extension
Two new research papers explore the MUON optimization algorithm, a method used in training large language models. The first paper introduces MALT, an extension of MUON that incorporates lightweight diagonal precondition…
-
New research outlines SGD preconditioner design for stability and noise reduction
A new research paper published on arXiv details design criteria for stochastic gradient descent (SGD) preconditioners, focusing on local conditioning, noise floors, and basin stability. The paper derives bounds where co…
-
Research paper details "benign misfitting" in linear regression models
A new research paper explores the phenomenon of "benign misfitting" in linear regression models, where a model that performs poorly on training data can still generalize well to new, unseen data. This occurs in a specif…
-
New framework offers stochastic saddle avoidance beyond unit excitation
A new research paper introduces a pathwise Lyapunov-Perron framework for stochastic saddle avoidance, moving beyond the traditional unit excitation assumption. This framework addresses scenarios where noise diminishes n…
-
New CAP-TTA framework enhances LLM debiasing for narrative generation
Researchers have developed CAP-TTA, a novel test-time adaptation framework designed to improve the debiasing of large language models (LLMs) when encountering out-of-distribution, high-bias prompts. This framework utili…
-
New research links mini-batch noise to loss landscape sharpness in SGD
A new research paper proposes that mini-batch noise during Stochastic Gradient Descent (SGD) training influences the sharpness of the loss landscape by causing fluctuations within the dominant subspace. The authors argu…
-
Seesaw method accelerates LLM training by optimizing batch size and learning rate
Researchers have developed a new method called Seesaw to accelerate the training of large language models by optimizing the scheduling of batch sizes and learning rates. This approach theoretically demonstrates an equiv…
-
SS--RS--GD Inequalities Resolved with AI Assistance
A new paper published on arXiv addresses the SS--RS--GD inequalities, a conjecture from COLT 2021 concerning the operators of single-shuffle SGD, random-reshuffle SGD, and gradient descent. The research demonstrates tha…
-
New technique improves DP-SGD accuracy using model curvature
Researchers have developed a new technique called NoiseCurve to improve the accuracy of differentially private stochastic gradient descent (DP-SGD). This method uses model curvature, estimated from unlabeled data, to en…
-
Weight-tied transformers: Recurrence as an algorithm
A new research paper explores the convergence properties of weight-tied looped transformers, investigating when these architectures effectively implement algorithms. The study introduces four key findings: a "budget law…
-
Harvard researchers unveil simple quadratic model predicting LLM pre-training dynamics
Researchers at Harvard University have developed a simple quadratic model that accurately predicts the optimization dynamics of large language models during pre-training. By applying Taylor's theorem to real neural netw…
-
New spectral cap method enhances LLM training by controlling weight matrix geometry
Researchers have proposed a new method called an "Isotropy-Preserving Spectral Cap" to improve the training of large language models (LLMs). This technique aims to control the internal geometry of weight matrices during…
-
New PAC-Bayesian Analysis Method for Overparameterized Models
Researchers have developed a new PAC-Bayesian analysis method for overparameterized models that accounts for continuous parameter symmetries. By performing analysis on the quotient predictor space, the method removes KL…
-
PPO learning stagnation solved by scaling to 1M environments
Researchers have identified a key reason for performance plateaus in Proximal Policy Optimization (PPO) algorithms, a common issue in deep reinforcement learning. They found that stagnation occurs not due to exploration…
-
Research paper analyzes loss landscape of two-layer ReLU networks
A new research paper published on arXiv explores the loss landscape of two-layer ReLU networks, focusing on the impact of width-dependent hyperparameters and L2 regularization. The study derives conditions under which g…
-
New research details SGD scaling limits at flat minima
A new paper explores the scaling limits of Stochastic Gradient Descent (SGD) when applied to convex objectives with flat minima. The research demonstrates that for such objectives, the behavior of SGD fundamentally chan…
-
New Geometric Framework Models Transformer Architecture Across Five LLMs
Researchers have developed a continuous geometric framework to model the Transformer architecture, translating its discrete algebraic operations into differential geometry and measure theory. This framework yields quant…