gradient descent
PulseAugur coverage of gradient descent — every cluster mentioning gradient descent across labs, papers, and developer communities, ranked by signal.
- affiliated with stochastic gradient descent 90%
- used by logistic regression model 90%
- used by Deep Neural Networks 80%
- instance of stochastic gradient descent 70%
- competes with stochastic gradient descent 70%
- used by artificial neural network 70%
- competes with RMSprop 70%
- used by Momentum 70%
- used by deep learning 70%
- instance of DagsHub 70%
- instance of alphaXiv 70%
- instance of cs.LG 70%
12 day(s) with sentiment data
-
New method accelerates deep neural network training for high-dimensional functions
Researchers have developed a novel method to accelerate the training of deep neural networks for high-dimensional functions. This approach integrates contextual features into the initial layer of the network, which are …
-
New research establishes lower bound for gradient descent acceleration
A new research paper published on arXiv introduces a theoretical lower bound for stepsize-based acceleration of gradient descent in smooth convex optimization. The study establishes a convergence rate of \(\\Omega(T^{-1…
-
New optimization theory unifies DNN convexity and smoothness
Researchers have introduced a novel optimization framework for deep neural networks (DNNs) that generalizes classical convexity and smoothness concepts. This new framework, termed $\mathcal{H}(\psi)$-convexity and $\mat…
-
New Gradient Descent Method Achieves Optimal Risk in Classification
Researchers have developed a new method using early stopping for gradient descent in classification tasks with overparameterized data. This technique aims to achieve minimax-optimal excess zero-one risk, particularly in…
-
Adam optimizer differs from gradient descent in factored models
A new research paper explores the differing behaviors of optimization algorithms like Adam and gradient descent when applied to factored models. The study reveals that while gradient descent is implicitly biased towards…
-
Logistic regression parameter vector weakly aligns with max-margin direction
Researchers have theoretically demonstrated that the parameter vector in logistic regression weakly aligns with the max-margin direction within a specific number of iterations. This early-stage alignment phenomenon, obs…
-
Gradient Descent Favors Low-Rank Solutions Over Adam in Factored Models
A new paper explores the differing behaviors of optimization algorithms, specifically gradient descent and Adam, when applied to factored matrix models. The research indicates that gradient descent inherently favors low…
-
Generalized Quadratic Gradient framework unifies optimization methods
Researchers have introduced Generalized Quadratic Gradient (GQG), a novel optimization framework that unifies and extends existing second-order optimization methods. GQG abstracts the core principles of Quadratic Gradie…
-
Muon optimizer shows promise in theoretical and practical neural network training
Two new research papers explore the Muon optimizer, an approach designed to better handle matrix-structured parameters in neural networks. The first paper introduces a matrix-aware geometry for Sharpness-Aware Minimizat…
-
DGM and PINN Algorithms Proven to Converge to PDE Solutions
Researchers have mathematically proven that the Deep Galerkin Method (DGM) and Physics Informed Neural Networks (PINNs) can reliably converge to the correct solution for a specific class of semi-linear partial different…
-
Anthropic CEO clarifies stance on open-weights AI models
Anthropic CEO Dario Amodei clarified the company's stance on open-weights models, stating that Anthropic has never advocated for banning them. He emphasized that open-weights models without dangerous capabilities are a …
-
New meta-learning algorithm tackles AI optimization challenges
Researchers have introduced a novel meta-learning algorithm called Greedy dynamical meta-learning, designed to overcome limitations of existing optimization methods for large AI models. The algorithm employs a two-loop …
-
Instruction Tuning Enhances LLM Performance with Task-Specific Fine-tuning
Instruction tuning is a key method for enhancing Large Language Models (LLMs) by fine-tuning them on specific tasks and instructions. This process improves the model's ability to understand and respond accurately to use…
-
Paper explains why relative positional encodings improve transformer generalization
A new paper proposes an optimization-based explanation for why transformers with relative positional encodings generalize better to longer sequences than those with absolute encodings. The research suggests that the imp…
-
AI Fundamentals: Understanding the Neuron and its Role in Neural Networks
This article explains the fundamental computational unit of artificial neural networks: the neuron. It details how neurons process numerical inputs, influenced by weights and a bias, and then apply an activation functio…
-
New method for multi-snapshot spike deconvolution detailed in arXiv paper
Researchers have published a paper on arXiv detailing a new method for multi-snapshot spike deconvolution. The study introduces the variable-projection formulation of spike deconvolution (VarProSD), which simplifies the…
-
Schedule-Free optimization methods achieve optimal convergence rates in nonconvex settings
A new paper explores the theoretical underpinnings of Schedule-Free optimization methods in nonconvex settings, which are common in machine learning. The research provides worst-case convergence rate analyses for Schedu…
-
Optimal learning rate scaling in deep networks depends on data, research finds
A new research note explores the dynamics of deep scalar linear networks, demonstrating that optimal learning rate scaling is data-dependent. The study shows that data-agnostic scaling rules falter across different netw…
-
Gradient Descent Theory Extended for Complex Minima and Vector Outputs
This paper extends the theory of gradient descent (GD) with large step sizes to more complex scenarios. It addresses overparameterized least-squares problems with vector-valued outputs and analyzes neighborhoods of mani…
-
New hybrid method accelerates MIONet training
Researchers have introduced a novel hybrid least squares/gradient descent (LSGD) method designed to accelerate the training of MIONets. This approach extends existing LSGD techniques used for DeepONets. The method treat…