Researchers have identified a stability phase diagram for the Adam optimizer, revealing how its two momentum timescales govern training instabilities. They discovered an approximately linear boundary in the $(\beta_1,\beta_2)$ plane that separates spiky from non-spiky dynamics, which is linked to the effective exponent of the loss function. This boundary helps explain loss spikes by connecting Adam's momentum timescales to the geometry of finite-scale superquadratic losses beyond the Hessian. AI
IMPACT Provides deeper understanding of neural network training instabilities, potentially leading to more robust optimization techniques.
RANK_REASON Academic paper detailing novel findings on optimizer dynamics. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
