SGD
PulseAugur coverage of SGD — every cluster mentioning SGD across labs, papers, and developer communities, ranked by signal.
- instance of stochastic gradient descent 95%
- instance of gradient descent 90%
- used by Neural tangent kernel 80%
- used by Gotit.pub 70%
- used by DagsHub 70%
- instance of DagsHub 70%
- instance of Gotit.pub 70%
- authored ScienceCast 70%
- instance of ScienceCast 70%
- competes with AdamW 70%
- used by stochastic gradient descent 70%
- competes with stochastic gradient descent 70%
11 day(s) with sentiment data
-
New research quantifies stochastic rounding impact on SGD training
A new arXiv paper explores the impact of stochastic rounding on the training of deep learning models using Stochastic Gradient Descent (SGD). The research provides theoretical bounds on prediction error for linear netwo…
-
Quantization Error Drives Language Model Training Instability
Researchers have identified a significant cause of instability during large language model pretraining: loss spikes. These spikes, often emerging late in training and termed "time-bomb spikes," are linked to the growth …
-
New analysis sharpens SCAFFOLD convergence rates in federated learning
Researchers have developed a new analysis for the SCAFFOLD algorithm in federated learning, focusing on how client data similarity impacts convergence rates. The study establishes linear dependence on Hessian similarity…
-
New FA-GC Method Enhances Federated Learning with Adaptive Optimizers
Researchers have introduced Federated Adaptive Gradient Correction (FA-GC), a new method designed to improve federated learning with adaptive optimizers like Adam. Unlike previous approaches that correct gradients only …
-
Language models can learn during inference, research finds
A new research paper explores the concept of language models learning during inference, a process termed dynamic evaluation. The study found that standard softmax-attention models, when allowed to perform gradient steps…
-
New UTA method optimizes learning rate decay for AI models
Researchers from the University of Texas at Austin have developed a new learning rate decay method called Universal Training Adaptive (UTA) LRD. This method aims to improve model training by being adaptive to the traini…
-
New research details multi-pass SGD learning dynamics in high-dimensional models
A new paper published on arXiv details the learning dynamics of multi-pass Stochastic Gradient Descent (SGD) in high-dimensional, non-convex multi-index models. The research develops an asymptotically exact characteriza…
-
New framework analyzes catastrophic forgetting in continual learning
Researchers have developed a new algorithmic stability framework to analyze catastrophic forgetting in continual learning. This framework decomposes forgetting into intra-task and cross-task generalization gaps, and emp…
-
New policy-based method reduces black-box simulator calls by 10x
Researchers have developed a new method to optimize black-box simulators, which are often non-differentiable and computationally expensive. This approach involves learning a policy that guides the training of a differen…
-
New theory strengthens gradient clipping guarantees for heavy-tailed noise
Researchers have developed new theoretical guarantees for gradient clipping in optimization algorithms, specifically for smooth convex optimization under affine heavy-tailed noise. The study focuses on clipped SGD (Clip…
-
New theory proves SGD convergence in deep linear networks
Researchers have established new theoretical convergence guarantees for stochastic gradient descent (SGD) in the context of deep linear neural networks. This work extends existing gradient descent results to SGD under r…
-
Research paper analyzes momentum's role in Polyak-Ruppert averaging
This research paper explores the effectiveness of momentum in Polyak--Ruppert averaging for stochastic gradient descent (SGD) and related optimization methods. The authors analyze how quickly finite runs converge to asy…
-
New SPIN optimizer tackles catastrophic forgetting in continual learning
Researchers have introduced SPIN, a novel optimizer designed to enhance online continual learning (OCL) by addressing catastrophic forgetting. SPIN explicitly separates plasticity and stability by interpolating between …
-
New arXiv Paper Analyzes SGD with Infinite-Variance Gradients
A new arXiv paper delves into the mathematical underpinnings of Stochastic Gradient Descent (SGD) when dealing with infinite-variance gradient noise. The research characterizes SGD's accuracy by analyzing exact first-or…
-
New SAGA method mitigates catastrophic forgetting in AI models
Researchers have developed a new method called Sparse Autoencoder Geometric Anchoring (SAGA) to combat catastrophic forgetting in continual learning for AI models. SAGA uses sparse autoencoders trained at task boundarie…
-
New research explores Corrective Power in pairwise AUC optimization
A new research paper explores the concept of "Corrective Power" in optimizing pairwise AUC (Area Under the Curve) for machine learning models, particularly in scenarios with imbalanced classes. The study identifies that…
-
New method accelerates AI model training by truncating rollouts
Researchers have developed Stochastic Truncated On-Policy Distillation (STOPD), a method to accelerate on-policy distillation (OPD) by randomly truncating student rollouts. This technique reduces computational costs by …
-
Optimizer choice impacts neural network training data privacy, study finds
A new research paper explores how different optimization algorithms used in training neural networks can impact the privacy of the training data. The study, which extends previous work on Karush-Kuhn-Tucker conditions, …
-
New research explores convergence rates and noise handling in Stochastic Gradient Descent variants
Multiple research papers published on arXiv explore advancements in Stochastic Gradient Descent (SGD) and its variants. One paper focuses on the statistical behavior of SGD with constant step sizes, proving convergence …
-
New theory explains constant-step SGD with momentum
Researchers have developed a new theoretical framework to analyze the behavior of constant-step stochastic gradient descent with momentum (SGDM). This framework treats the coupled parameter and momentum recursions as a …