New method probes neural network loss sharpness for stable learning rates

Researchers have developed a novel method to estimate the local sharpness of a loss function in neural networks, a critical factor for stable gradient steps. By analyzing the step size accepted during Armijo backtracking line searches, they can derive a low-cost probe for the top Hessian eigenvalue. This probe, when used once during initialization, provides a learning rate safeguard that makes optimizers like Adam and AdamW robust to excessively large initial learning rates across a wide range of values and architectures. AI

IMPACT This research offers a low-cost method to improve the stability of training neural networks by safeguarding learning rates.

RANK_REASON The item is an academic paper detailing a new method for analyzing neural network loss functions. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Adam
• AdamW
• Armijo backtracking
• CIFAR-10
• Fashion-MNIST
• Imagenette

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →