stochastic gradient descent
PulseAugur coverage of stochastic gradient descent — every cluster mentioning stochastic gradient descent across labs, papers, and developer communities, ranked by signal.
- instance of Influence Flower 90%
- instance of Gotit.pub 70%
- instance of alphaXiv 70%
- instance of ScienceCast 70%
- competes with Adam optimizer 70%
- used by Deep Neural Networks 70%
- used by partial differential equations 70%
- instance of Deep Neural Networks 70%
- used by Neural tangent kernel 70%
- used by Gaussian Processes 70%
- used by Influence Flower 60%
- used by Adam optimizer 50%
6 day(s) with sentiment data
-
New optimization theory unifies DNN convexity and smoothness
Researchers have introduced a novel optimization framework for deep neural networks (DNNs) that generalizes classical convexity and smoothness concepts. This new framework, termed $\mathcal{H}(\psi)$-convexity and $\mat…
-
Weak correlations principle explains linearization in gradient-based learning systems
A new paper published on arXiv explores the principle of weak correlations as the underlying reason for the linearization observed in gradient-based learning systems. The research suggests that the simplified dynamics s…
-
Researchers Analyze Stochastic Gradient Descent with Discontinuities
Researchers have analyzed stochastic gradient descent (SGD) when applied to loss functions that exhibit discontinuity across lower-dimensional manifolds. The study focuses on the differential equation limit of SGD to un…
-
New research outlines SGD preconditioner design for stability and noise reduction
A new research paper published on arXiv details design criteria for stochastic gradient descent (SGD) preconditioners, focusing on local conditioning, noise floors, and basin stability. The paper derives bounds where co…
-
Research paper details "benign misfitting" in linear regression models
A new research paper explores the phenomenon of "benign misfitting" in linear regression models, where a model that performs poorly on training data can still generalize well to new, unseen data. This occurs in a specif…
-
New research explores Compactly Supported Radial Basis Functions for probability density modeling
Researchers have explored the use of Compactly Supported Radial Basis Functions (CS-RBFs) as a novel parametric family for probability density functions, particularly focusing on Wendland $\mathscr{C}^2$ kernels. The st…
-
New method accurately attributes data contributions for Adam optimizer
Researchers have developed a new method called Adam-Aware In-Run Data Shapley to accurately attribute data contributions in machine learning models that use the Adam optimizer. Existing methods, which rely on the linear…
-
Research paper details how learning-rate cooldown effectiveness depends on noise and optimizer normalization
A new research paper explores the effectiveness of the learning-rate cooldown phase in large-model pretraining, a common component of Warmup-Stable-Decay (WSD) schedules. The study reveals that the benefit of this coold…
-
New algorithm CREATE-IF-LATE counters "Pac-Man attack" in decentralized learning
Researchers have developed a new algorithm called CREATE-IF-LATE (CIL) to address a novel threat in decentralized learning known as the "Pac-Man attack." This attack involves a malicious node probabilistically terminati…
-
New framework precisely models multi-class SGD dynamics in high dimensions
Researchers have developed a new framework to analyze the learning dynamics of multi-class stochastic gradient descent (SGD) in high-dimensional settings. The framework provides exact expressions for key metrics like ri…
-
New ETBQ method boosts low-bit neural network quantization accuracy
Researchers have developed a new method called Efficient Tuning Before Quantization (ETBQ) to improve the accuracy of low-bit post-training quantization (PTQ) for deep neural networks. This technique involves a pre-cond…
-
Schedule-Free optimization methods achieve optimal convergence rates in nonconvex settings
A new paper explores the theoretical underpinnings of Schedule-Free optimization methods in nonconvex settings, which are common in machine learning. The research provides worst-case convergence rate analyses for Schedu…
-
Bayesian teachers enhance AI model distillation accuracy and stability
A new research paper explores knowledge distillation (KD) through a Bayesian lens, analyzing student model convergence with Stochastic Gradient Descent (SGD). The study reveals that using Bayesian deep learning models a…
-
New EISAM optimizer enhances deep learning generalization
Researchers have introduced Extragradient-Inspired Sharpness-Aware Minimization (EISAM), a new optimizer designed to improve generalization in deep learning. EISAM employs a two-step process, involving a prediction and …
-
New adaptive Adam optimizer improves deep learning convergence for PDEs
A new paper introduces a learning-rate-adaptive variant of the Adam optimizer designed to improve convergence in deep learning, particularly for solving partial differential equations. The proposed method adjusts the le…
-
New thermodynamic framework models neural network training like ideal gas behavior
Researchers have developed a thermodynamic framework to analyze the training dynamics of scale-invariant neural networks trained with stochastic gradient descent (SGD). This framework draws parallels between training hy…
-
New research guarantees convergence for physics-informed neural networks trained with SGD
Researchers have established the linear convergence of stochastic gradient descent (SGD) for training over-parameterized two-layer physics-informed neural networks (PINNs) when solving the Poisson equation. This analysi…
-
Paper analyzes SGD convergence for score-based generative models
Researchers have published a paper analyzing the optimization dynamics of training Score-based Generative Models (SGMs). The study focuses on Stochastic Gradient Descent (SGD) and provides convergence rates for general …
-
New research details SGD convergence for score-based generative models
Researchers have published a paper detailing the non-asymptotic convergence of Stochastic Gradient Descent (SGD) when applied to Score-based Generative Models (SGMs). The study provides theoretical guarantees for SGD in…
-
LoRA technique enables efficient fine-tuning of large AI models
Several articles discuss fine-tuning large language models, with a particular focus on the LoRA (Low-Rank Adaptation) technique. LoRA allows for efficient adaptation of large models by training only a small fraction of …