Hessian
PulseAugur coverage of Hessian — every cluster mentioning Hessian across labs, papers, and developer communities, ranked by signal.
5 day(s) with sentiment data
-
New research unifies models for neural network stability dynamics · 2 sources tracked
Two new arXiv papers explore the complex dynamics of neural networks at the edge of stability. The first paper, "Symmetry Breaking at the Edge of Stability via Central Flow Analysis," by Naima Elosegui Borras and Marcot…
-
New equilibrium neural networks offer rigorous bounds for Monge--Ampère equation
Researchers have developed a new class of equilibrium neural networks, specifically nonnegative softplus networks with a contraction factor q<1, designed to handle the Monge--Ampère equation. This approach addresses cha…
-
New Theory Explains Orthogonalization's Effectiveness in Neural Networks
Researchers have developed a new theoretical framework to explain why orthogonalization techniques, such as Muon, are effective in optimizing neural networks. The study reveals that the Hessian of a neural network, whic…
-
New Target-Response Jacobian metric outperforms influence measures in AI model analysis
Researchers have introduced the target-response Jacobian ($J$), a new metric for understanding how model predictions change in response to alterations in individual training targets. This metric complements existing tra…
-
New arXiv Paper Analyzes SGD with Infinite-Variance Gradients
A new arXiv paper delves into the mathematical underpinnings of Stochastic Gradient Descent (SGD) when dealing with infinite-variance gradient noise. The research characterizes SGD's accuracy by analyzing exact first-or…
-
New framework Controlled-Minimax stabilizes adversarial optimization
Researchers have developed a new framework called Controlled-Minimax to address the stability issues inherent in adversarial optimization. By modeling training dynamics as a second-order Ordinary Differential Equation (…
-
New JAX toolkit enables efficient Automatic Sparse Differentiation
A new toolkit called asdex has been released for the JAX ecosystem, offering the first standalone implementation of Automatic Sparse Differentiation (ASD). This method is designed to efficiently compute Jacobian or Hess…
-
Deep learning Hessian spectra explained by hidden symmetry
A new paper published on arXiv explores the spectral properties of the Hessian matrix in deep learning models. Researchers have observed that eigenvalues in trained models tend to cluster, with a large group near zero a…
-
New research questions Hessian trace as reliable model-selection signal
A new research paper explores the use of internal model properties, specifically the Hessian trace and its eigenvalues, as a proxy for model selection when external validation data is unreliable. The study found that wh…
-
New research details quantization failures in looped transformers
Researchers have identified two critical failure modes in post-training quantization (PTQ) for looped transformers, which reuse weights across recurrence steps. The first, termed 'feedback exposure,' occurs when a quant…
-
New algorithm simplifies federated bilevel optimization using first-order gradients
Researchers have developed a new algorithm for federated stochastic bilevel optimization that avoids the need for computationally expensive second-order Hessian and Jacobian matrices. This novel approach, termed federat…
-
New research explores geometric optimization at the 'Edge of Stability' in associative memories
Researchers have explored the geometric properties of high-capacity kernel logistic regression (KLR) associative memories, identifying a critical hyperparameter regime known as the "Ridge of Optimization." This region i…
-
New Newton's Method Achieves O(1/k^3) Convergence Rate
Researchers have developed a novel direct accelerated Newton method for minimizing convex functions with Lipschitz continuous Hessians. This new algorithm operates solely with primal variables and requires only one line…
-
New 'Rényi Sharpness' metric shows strong generalization correlation
Researchers have introduced "Rényi sharpness," a new metric for evaluating neural network generalization that aims to improve upon existing methods. Unlike traditional sharpness measures that focus on average loss or ma…
-
New theory reframes optimizer stability in deep learning
Researchers have identified a phenomenon in deep learning where gradient-based optimizers maintain stable Hessian eigenvalues above theoretically predicted instability thresholds. This deviation, observed to be as high …
-
New theoretical bounds improve Langevin sampling for complex distributions
Researchers have developed new theoretical bounds for the Moreau--Yosida unadjusted Langevin algorithm (MYULA), a method used for sampling from complex probability distributions. The study focuses on nonsmooth composite…
-
New theory explains curvature in Probabilistic Circuits
Researchers have developed a compositional theory for understanding curvature in Probabilistic Circuits (PCs), a type of generative model. They demonstrated that the Hessian trace, a measure of loss-surface curvature, c…
-
Generalized Quadratic Gradient framework unifies optimization methods
Researchers have introduced Generalized Quadratic Gradient (GQG), a novel optimization framework that unifies and extends existing second-order optimization methods. GQG abstracts the core principles of Quadratic Gradie…
-
New research links mini-batch noise to loss landscape sharpness in SGD
A new research paper proposes that mini-batch noise during Stochastic Gradient Descent (SGD) training influences the sharpness of the loss landscape by causing fluctuations within the dominant subspace. The authors argu…
-
New kinetic theory formalizes zeroth-order Newton methods
Researchers have developed a formal kinetic theory for zeroth-order Newton-type methods, which are useful when gradients and Hessians are unavailable. The framework includes a Gaussian-Stein correction to accurately est…