gradient descent
PulseAugur coverage of gradient descent — every cluster mentioning gradient descent across labs, papers, and developer communities, ranked by signal.
- used by Deep Neural Networks 90%
- instance of stochastic gradient descent 90%
- used by logistic regression model 90%
- used by artificial neural network 80%
- affiliated with stochastic gradient descent 70%
- instance of Deep Neural Networks 70%
- used by AdamW 70%
- instance of Influence Flower 70%
- used by Hessian 70%
- used by deep learning 70%
- competes with stochastic gradient descent 70%
- instance of CatalyzeX 70%
13 day(s) with sentiment data
-
Language models can now "train" neural networks by editing weights
Researchers have demonstrated a novel technique called weight-space programming, which allows language models to directly edit the weights of trained neural networks. This method enables "training" through reasoning rat…
-
New research explores adversarial training for AI robustness · 2 sources tracked
Two new arXiv papers explore adversarial training techniques for improving AI model robustness. The first paper analyzes the dynamics of gradient descent with large step sizes in adversarial linear logistic regression a…
-
New research unifies models for neural network stability dynamics · 2 sources tracked
Two new arXiv papers explore the complex dynamics of neural networks at the edge of stability. The first paper, "Symmetry Breaking at the Edge of Stability via Central Flow Analysis," by Naima Elosegui Borras and Marcot…
-
Multi-UAV Cooperative Deep Semantic Autoencoders enhance drone network communication
Researchers have developed a novel system called Multi-UAV Cooperative Deep Semantic Autoencoders (MC-DSA) to improve communication efficiency and data processing in drone networks. This system utilizes a Task UAV (T-UA…
-
Muon optimizer research reveals complex pretraining dynamics and fine-tuning trade-offs
Recent research explores the performance of the Muon optimizer in large language model pretraining, revealing nuanced behaviors that differ from classical gradient descent theories. One study indicates that Muon's dynam…
-
New research details gradient descent dynamics in simplified neural networks
A new paper published on arXiv details the complex dynamics of gradient descent in a simplified neural network model. Researchers analyzed a two-neuron Rectified Linear Unit (ReLU) network trained with gradient flow, re…
-
AutoHebb networks mimic human learning with self-modifying neural architecture · 2 sources tracked
Researchers have introduced AutoHebb, a novel class of self-modifying neural networks designed to mimic human and animal learning capabilities like zero-shot inference and continual learning. Unlike traditional models t…
-
New PGD algorithm sharpens neural network regression rates
Researchers have developed a new Preconditioned Gradient Descent (PGD) algorithm for training over-parameterized neural networks in nonparametric regression tasks. This method, combined with early stopping, achieves a s…
-
AI models cannot fully unlearn user data, posing privacy challenges
The concept of "AI having your data" is complex, encompassing training data, memorized content within model weights, and inference logs. While companies can delete inference logs upon account deletion, they cannot easil…
-
Research paper analyzes Muon optimizer's PolarGD update method
A new research paper published on arXiv analyzes the "PolarGD" update method, a core component of the Muon optimizer. The study, which uses matrix quadratics for analysis, suggests that PolarGD's orthogonalization strat…
-
Research paper details phase transitions in machine learning with extremal data
A new research paper published on arXiv explores phase transitions in learning, specifically within extremal regression problems that are a type of multiple instance learning (MIL). The study focuses on how max pooling,…
-
Gradient Descent on Logistic Regression: Equal Norms and Convergence
A new research paper explores the behavior of gradient descent on logistic regression models, particularly when data points are normalized to have equal norms. The study proves that for linearly separable data, gradient…
-
New theory proves SGD convergence in deep linear networks
Researchers have established new theoretical convergence guarantees for stochastic gradient descent (SGD) in the context of deep linear neural networks. This work extends existing gradient descent results to SGD under r…
-
New research explores ReLU networks beyond linear separability
Researchers have analyzed a simplified neural network model with ReLU activation functions and logistic loss on non-linearly separable data. They identified an 'active cone' in parameter space where perfect classifiers …
-
New research explores if removing AI model parts can boost performance
Researchers have explored a phenomenon called Capability Enhancement Localization (CEL) in transformer-based language models. This concept investigates whether removing a small subset of attention heads can actually imp…
-
New framework explains why neural networks learn low-frequency components first
A new theoretical framework, the Orthogonal Polynomial Dynamics Framework, has been developed to explain why shallow neural networks prioritize learning low-frequency components, a phenomenon known as spectral bias. Thi…
-
New research tackles catastrophic forgetting in LLMs with novel tuning and regularization methods · 6 sources tracked
Researchers are exploring new methods to address catastrophic forgetting in large language models (LLMs) during continual learning. One approach, Subspace Tuning (SST), restricts weight updates to specific subspaces to …
-
New Test-Time Forecasting framework enables foundation models to predict unseen dynamics
Researchers have introduced a new framework called Test-Time Forecasting (TTF) that enables foundation models to predict the dynamics of unseen dynamical systems. This framework utilizes looped transformers and gradient…
-
One-layer transformers can learn K-NN prediction with Chain-of-Thought, study finds
A new research paper explores how one-layer transformers can learn in-context K-nearest neighbor prediction using Chain-of-Thought (CoT) reasoning. While previous work showed one-layer transformers could perform 1-NN pr…
-
New research probes Muon optimizer's geometry, performance, and limitations · 4 sources tracked
Four recent arXiv papers explore the optimization algorithm Muon, investigating its theoretical underpinnings and practical performance. The first paper delves into the geometry of Muon, introducing the Random-Radius Is…