gradient descent
PulseAugur coverage of gradient descent — every cluster mentioning gradient descent across labs, papers, and developer communities, ranked by signal.
- used by Deep Neural Networks 90%
- affiliated with stochastic gradient descent 90%
- used by logistic regression model 90%
- used by deep learning 90%
- used by rectifier 90%
- used by artificial neural network 80%
- instance of Influence Flower 70%
- competes with stochastic gradient descent 70%
- instance of stochastic gradient descent 70%
- used by Hessian 70%
- used by Gotit.pub 70%
- instance of Deep Neural Networks 70%
9 day(s) with sentiment data
-
Research paper details how gradient descent amplifies bias in ML models
A new research paper introduces a formal framework to understand how gradient descent can amplify biases in machine learning models, particularly affecting minority groups. The study, illustrated with deep learning expe…
-
Transformer models need position embedding to understand word order
This technical article explains the necessity of position embedding in Transformer models by building a simplified, hand-constructed version. The author demonstrates how word order becomes significant when introducing a…
-
New research explores geometric optimization at the 'Edge of Stability' in associative memories
Researchers have explored the geometric properties of high-capacity kernel logistic regression (KLR) associative memories, identifying a critical hyperparameter regime known as the "Ridge of Optimization." This region i…
-
Paper contrasts Gradient Descent and Natural Gradient Descent on KLR-trained Hopfield networks
A new paper analyzes the geometry of learning dynamics in high-capacity associative memories, specifically Kernel Logistic Regression (KLR) trained Hopfield networks. It compares Gradient Descent (GD) and Natural Gradie…
-
New framework unifies first-order optimization inequalities for statistical analysis
A new paper introduces "basic inequalities" for first-order optimization algorithms, providing a framework that connects implicit and explicit regularization. This framework bounds the objective function's difference fr…
-
New research explores advanced gradient descent for operator learning and optimization
Two new research papers explore advanced gradient descent techniques for complex optimization problems. The first paper details stochastic gradient descent (SGD) for learning operators between Hilbert spaces, establishi…
-
New method improves Kernel PCA for streaming data
Researchers have developed a new method for Kernel Principal Component Analysis (KPCA) designed to handle streaming data and adapt to changes over time. This rotation-based subspace tracking approach updates the model b…
-
Tree Tensor Networks Reveal Benign Loss Landscapes Despite Hard Targets
Researchers have explored the theoretical underpinnings of why deep neural networks, despite their complexity, often learn effectively in practice. A new study using Tree Tensor Networks (TTNs) demonstrates that even mo…
-
New theory explains incremental learning in shallow neural networks
Researchers have developed a new theoretical framework for understanding incremental learning in shallow neural networks. This work focuses on polynomial-width two-layer networks trained on orthogonal multi-index target…
-
New algorithm enhances distributed gradient descent generalization
This paper analyzes the generalization capabilities of distributed gradient descent algorithms within a reproducing kernel Hilbert space. Researchers developed the Distributed Kernel-based Robust Gradient Descent (DKRGD…
-
Z-transform method applied to quadratic optimization in new research paper
A new paper explores the application of the z-transform method to quadratic optimization problems. The research demonstrates how this classical tool, typically used in signal processing and control theory, can yield nov…
-
New research explores theoretical limits of neural network generalization · 4 papers
Four new research papers delve into the theoretical underpinnings of generalization in neural networks. One paper establishes a necessary and sufficient condition for provable compositional generalization, focusing on s…
-
New research advances stochastic optimization for machine learning · 5 sources tracked
Several recent research papers explore advancements in stochastic optimization techniques, particularly focusing on gradient descent and its variants for complex machine learning problems. One paper demonstrates that va…
-
New paper proposes outcome-indexed attention matrices to fix learning instability
Researchers have identified a critical instability in how attention vectors are implemented in machine learning models, particularly when dealing with multiple outcomes. The standard approach of using a globally shared …
-
AI training fundamentals: Gradient descent and AdamW explained
Two articles from Towards AI delve into the fundamental concepts of machine learning training. The first article explains the limitations of gradients in training large models, highlighting the necessity of optimizers l…
-
New sparse data augmentation method offers provable optimization guarantees
Researchers have developed a new method for sparse data augmentation in nonconvex optimization problems, particularly relevant for geometric machine learning. This technique allows for the approximation of full data aug…
-
Gradient Descent Dynamics Explored in New Optimization Research
Two new arXiv papers delve into the complexities of gradient descent algorithms. The first paper by Si Yi Meng examines gradient descent dynamics on logistic regression with non-separable data and large step sizes, reve…
-
New theory explains gradient descent dynamics at edge of stability
Researchers have developed a new perturbative approach to formally derive the central flow model of gradient descent at the edge of stability in deep learning. This method treats gradient descent as a singularly perturb…
-
New research reveals divergence in ReLU neural network training dynamics
A new paper published on arXiv explores the mathematical underpinnings of training neural networks with ReLU activation functions. The research demonstrates that the gradient descent algorithm, when applied to these net…
-
New early stopping rule for neural networks bypasses training
Researchers have developed a new data-dependent early stopping rule for training neural networks that estimates generalization error analytically, bypassing the need for numerical estimation through gradient descent. Th…