stochastic gradient descent
PulseAugur coverage of stochastic gradient descent — every cluster mentioning stochastic gradient descent across labs, papers, and developer communities, ranked by signal.
6 天有情绪数据
-
New paper re-evaluates SGD dynamics, challenging Brownian motion analogy
A new paper challenges the common assumption that Stochastic Gradient Descent (SGD) noise behaves like Brownian motion. Researchers propose an alternative model where SGD dynamics occur within a fluctuating loss landsca…
-
Simple Random Node Sampling outperforms full-graph training for GNNs
Researchers have found that a simple Random Node Sampling (RNS) method for training Graph Neural Networks (GNNs) can match or exceed the performance of full-graph training. This surprising result holds true across numer…
-
New Bayesian Framework Optimizes Neural Network Learning Rates
Researchers have introduced a novel probabilistic framework to optimize the learning rate in neural network training, moving beyond empirical trial-and-error. This new approach develops classic Bayesian statistics into …
-
New method adds missingness to SGD to reduce bias in incomplete data
Researchers have developed a novel method called Richardson-SGD to address gradient bias in stochastic gradient descent when dealing with incomplete data. The technique involves deliberately introducing additional missi…
-
Factor Augmented SGD optimizes high-dimensional machine learning
Researchers have introduced Factor-Augmented SGD (FSGD), a novel optimization method designed for high-dimensional machine learning tasks. FSGD operates on streaming data, enabling scalability for large-scale problems w…
-
Adam optimizer corrects SGD's frequency bias in language model training
New research highlights a frequency bias in Stochastic Gradient Descent (SGD) when training language models on imbalanced token distributions. This bias causes parameters for common tokens to converge quickly, while tho…
-
New theory shows momentum enables perfect parallelization in SGD
Researchers have developed a new theory explaining how classical momentum schemes like Polyak's heavy ball can accelerate stochastic gradient descent (SGD) for large-scale machine learning. The theory applies to quadrat…
-
LLM training research explores distillation, feedback, and optimizers
New research explores methods to improve Large Language Model (LLM) training efficiency and effectiveness. One study challenges the necessity of a strong teacher model in knowledge distillation, finding that even smalle…
-
Paper analyzes SGD dynamics in high-dimensional linear networks
A new paper details the high-dimensional behavior of stochastic gradient descent (SGD) on diagonal linear networks. The research shows that in high dimensions, SGD dynamics can be accurately modeled by a stochastic diff…
-
New papers analyze gradient descent convergence in neural networks
Two new research papers explore the convergence properties of gradient descent in neural network training. The first paper, focusing on wide shallow models with bounded nonlinearities, proves that non-global minimizers …
-
New method tackles unbounded variance in variational inference
Researchers have developed a new approach to optimize Black-Box Variational Inference (BBVI) by addressing the inherent unbounded variance in its stochastic gradients. Their method, detailed in a new paper, focuses on t…
-
New research derives advanced optimizers from evolutionary principles
Researchers have developed a new method to derive advanced optimization algorithms directly from evolutionary principles, unifying previously disparate views of evolution. This approach introduces Darwinian Lineage Simu…
-
Bayesian Parameter Shift Rule enhances VQE gradient estimation
Researchers have introduced a Bayesian variant of the parameter shift rule (PSR) for variational quantum eigensolvers (VQEs). This new method utilizes Gaussian processes to estimate objective function gradients, offerin…
-
Researchers explore efficient parameter estimation for truncated Boolean product distributions
Researchers have developed a new method for estimating parameters of truncated Boolean product distributions, a problem previously unaddressed in discrete settings. The approach relies on a concept of 'fatness' for the …
-
Researchers develop novel bootstrap for SGD confidence sets
Researchers have developed a novel method for constructing confidence sets in Stochastic Gradient Descent (SGD) algorithms. This new approach utilizes the multiplier bootstrap procedure and establishes its non-asymptoti…
-
Evolutionary game theory deciphers shortcut learning in deep neural networks
Researchers have developed a new theoretical framework using evolutionary game theory to understand shortcut learning in deep neural networks. The study formally defines core and shortcut features, modeling data samples…
-
New method bridges graph drawing and dimensionality reduction using stochastic optimization
Researchers have developed a new method that bridges graph drawing and dimensionality reduction techniques by adapting stochastic gradient descent for vector data embedding. This approach, implemented as a scikit-learn …
-
Researchers explore complex SGD and directional bias in kernel Hilbert spaces
Researchers have introduced a novel variant of Stochastic Gradient Descent (SGD) designed for complex-valued neural networks. This new method, termed complex SGD, offers convergence guarantees even without analyticity c…
-
Researchers develop SGD algorithms for learning operators with operator-valued kernels
Researchers have developed a new method for estimating regression operators in statistical inverse problems. The approach utilizes regularized stochastic gradient descent (SGD) with operator-valued kernels, offering dim…
-
New research refines SGD generalization bounds and covariance estimation
Researchers have developed new methods to analyze the generalization capabilities of Stochastic Gradient Descent (SGD) in machine learning. One paper introduces predictable history-adaptive virtual perturbations, allowi…