New research links mini-batch noise to loss landscape sharpness in SGD

A new research paper proposes that mini-batch noise during Stochastic Gradient Descent (SGD) training influences the sharpness of the loss landscape by causing fluctuations within the dominant subspace. The authors argue that this dominant subspace, typically defined by the top eigenvectors of the Hessian, is crucial for understanding sharpness dynamics rather than direct loss reduction. Their work derives a sharpness correction term related to mini-batch noise in these dominant directions, and experiments show that incorporating this term brings Gradient Descent's sharpness evolution closer to that of SGD. AI

IMPACT Provides a theoretical framework for understanding how mini-batch noise affects model training, potentially leading to more stable and efficient optimization techniques.

RANK_REASON Academic paper published on arXiv detailing a new theoretical insight into SGD training dynamics. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• alphaXiv
• arXiv
• CatalyzeX Code Finder for Papers
• CORE Recommender
• DagsHub
• Gotit.pub
• Hessian
• Hugging Face
• IArxiv Recommender
• Influence Flower
• ScienceCast
• SGD

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →