New equations quantify learning-rate transfer curve gradients in neural networks

Researchers have developed new methodologies to quantitatively characterize the shape of learning-rate (LR) transfer curves in neural networks. They propose two relational equations that empirically explain the gradient of the LR transfer curve, referred to as the LR gradient. One equation is a theoretical formula applicable at the edge of stability, while the other is an approximate expression derived by treating the gradient as a hyper-parameter gradient. These equations can establish a lower bound on the curve's gradient and identify conditions for a zero gradient in normalization layers, offering a foundation for understanding LR dependence. AI

IMPACT Provides a theoretical foundation for understanding and optimizing neural network training dynamics.

RANK_REASON The cluster contains an academic paper detailing new theoretical and empirical findings in machine learning. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• Hugging Face
• learning-rate transfer curves
• LR curve
• LR gradient
• LR transfer
• machine learning
• neural networks
• normalization layers

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →