A new theoretical perspective on learning rate warm-up in deep learning has been proposed, explaining its effectiveness in improving training. Researchers have generalized the $(L_0, L_1)$-smoothness condition to bound local curvature based on loss suboptimality. This condition, satisfied by common neural architectures, accurately reflects the optimization landscape early in training. Adapting the learning rate to this curvature naturally leads to a warm-up schedule, offering provably faster convergence than fixed learning rates, as demonstrated by experiments on language and vision models. AI
IMPACT Provides a theoretical foundation for a common deep learning heuristic, potentially leading to more robust and efficient training practices.
RANK_REASON The item is a research paper published on arXiv detailing theoretical and empirical findings on a machine learning optimization technique. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
