stochastic gradient descent
PulseAugur coverage of stochastic gradient descent — every cluster mentioning stochastic gradient descent across labs, papers, and developer communities, ranked by signal.
10 day(s) with sentiment data
-
New HybridSGD method optimizes distributed-memory AI training
Researchers have developed HybridSGD, a novel 2D parallel stochastic gradient descent method designed to optimize performance in distributed-memory systems. This new approach offers a continuous trade-off between existi…
-
New theory links shock-wave dynamics to neural network training
Researchers have established a mathematical connection between shock-wave theory and the learning dynamics of stochastic gradient descent in artificial neural networks. By applying principles from differential geometry,…
-
Machine Learning in Healthcare Course Syllabus Detailed
This document outlines a comprehensive curriculum for a Machine Learning in Healthcare course. It covers fundamental concepts like the distinction between machine learning and deep learning, various neural network archi…
-
Dropout in Neural Networks Linked to Percolation Theory
A new research paper explores the concept of percolation within neural networks trained using dropout regularization. The study, submitted to arXiv, posits that the random removal of connections during dropout training …
-
New Theory: SA-Adam Adaptivity Asymptotically Invisible
Researchers have published a paper detailing a theoretical analysis of adaptive optimization algorithms, specifically focusing on SA-Adam with momentum and non-convergent adaptive preconditioning. The study proves a non…
-
New optimization techniques emerge for faster, more efficient AI model training · 8 sources tracked
Several recent arXiv papers explore advancements in optimization techniques for machine learning. Researchers have proposed new methods like Weight Adaptation ASNG (WA-ASNG) to improve parallel performance in evolutiona…
-
Gradient Descent Outperforms Ridge Regression in Linear Models
A new research paper published on arXiv analyzes the performance of gradient descent (GD) compared to ridge regression and online stochastic gradient descent (SGD) in linear regression tasks. The study finds that GD con…
-
New study reveals SGD noise-covariance link to loss landscape curvature
Researchers have uncovered a new relationship between the noise introduced by Stochastic Gradient Descent (SGD) and the curvature of the loss landscape in deep learning models. Their findings indicate that this noise is…
-
New method predicts neural network generalization using Fourier fractal dimension
Researchers have developed a new method to predict how well deep neural networks will generalize without needing separate validation data. This approach uses the Fourier fractal dimension of the network's weight variati…
-
Deep Neural Networks Achieve Optimal Generalization Rates
Two new papers submitted to arXiv analyze the generalization performance of gradient descent methods in deep neural networks. The research establishes minimax-optimal rates for excess population risk in deep ReLU networ…
-
Lyapunov framework analyzes stochastic algorithm convergence
Researchers have published a paper detailing a Lyapunov-based framework for analyzing the finite-time convergence of stochastic iterative algorithms. This approach uses generalized Moreau envelopes as universal Lyapunov…
-
New research shows implicit regularization enhances AI attribution robustness
Researchers have demonstrated that adversarial robustness in deep learning attributions can emerge implicitly through standard stochastic gradient descent, negating the need for computationally intensive explicit regula…
-
New paper re-evaluates SGD dynamics, challenging Brownian motion analogy
A new paper challenges the common assumption that Stochastic Gradient Descent (SGD) noise behaves like Brownian motion. Researchers propose an alternative model where SGD dynamics occur within a fluctuating loss landsca…
-
Simple Random Node Sampling outperforms full-graph training for GNNs
Researchers have found that a simple Random Node Sampling (RNS) method for training Graph Neural Networks (GNNs) can match or exceed the performance of full-graph training. This surprising result holds true across numer…
-
New Bayesian Framework Optimizes Neural Network Learning Rates
Researchers have introduced a novel probabilistic framework to optimize the learning rate in neural network training, moving beyond empirical trial-and-error. This new approach develops classic Bayesian statistics into …
-
New method adds missingness to SGD to reduce bias in incomplete data
Researchers have developed a novel method called Richardson-SGD to address gradient bias in stochastic gradient descent when dealing with incomplete data. The technique involves deliberately introducing additional missi…
-
Factor Augmented SGD optimizes high-dimensional machine learning
Researchers have introduced Factor-Augmented SGD (FSGD), a novel optimization method designed for high-dimensional machine learning tasks. FSGD operates on streaming data, enabling scalability for large-scale problems w…
-
Adam optimizer corrects SGD's frequency bias in language model training
New research highlights a frequency bias in Stochastic Gradient Descent (SGD) when training language models on imbalanced token distributions. This bias causes parameters for common tokens to converge quickly, while tho…
-
New theory shows momentum enables perfect parallelization in SGD
Researchers have developed a new theory explaining how classical momentum schemes like Polyak's heavy ball can accelerate stochastic gradient descent (SGD) for large-scale machine learning. The theory applies to quadrat…
-
LLM training research explores distillation, feedback, and optimizers
New research explores methods to improve Large Language Model (LLM) training efficiency and effectiveness. One study challenges the necessity of a strong teacher model in knowledge distillation, finding that even smalle…