PulseAugur
EN
LIVE 07:19:16

Hessian Eigenvector Dynamics Reveal Optimizer Differences in Neural Network Training

By PulseAugur Editorial · [2 sources] ·

Researchers have analyzed the evolution of Hessian eigenvectors during neural network training, revealing distinct behaviors between different optimizers. The study found that SGD tends to stabilize leading curvature directions over time, while Adam shows significant reorganization of these eigenvectors. Additionally, Adam exhibits a localization phenomenon where a small set of parameters disproportionately influences the leading curvature. AI

IMPACT Provides deeper insights into how optimizers like SGD and Adam affect neural network training, potentially guiding future algorithm development.

RANK_REASON The cluster contains an academic paper detailing novel research findings on neural network training dynamics.

Read on arXiv cs.LG →

AI-generated summary · Google Gemini · from 2 sources. How we write summaries →

Hessian Eigenvector Dynamics Reveal Optimizer Differences in Neural Network Training

COVERAGE [2]

  1. arXiv cs.LG TIER_1 English(EN) · Marcelina Marjankowska, Valerio Modugno, Paolo Barucca ·

    Characterizing Optimizer-Dependent Training Dynamics Through Hessian Eigenvector Displacement and Localization

    arXiv:2606.30226v1 Announce Type: new Abstract: Hessian spectral properties are a standard tool in analysing neural-network training, with eigenvalues linked to sharpness, generalization, and optimization dynamics. Eigenvalues quantify curvature magnitude, while eigenvectors iden…

  2. arXiv cs.LG TIER_1 English(EN) · Paolo Barucca ·

    Characterizing Optimizer-Dependent Training Dynamics Through Hessian Eigenvector Displacement and Localization

    Hessian spectral properties are a standard tool in analysing neural-network training, with eigenvalues linked to sharpness, generalization, and optimization dynamics. Eigenvalues quantify curvature magnitude, while eigenvectors identify which parameters generate that curvature. I…

RELATED ENTITIES

RELATED TOPICS