On the Stability of Nonlinear Dynamics in GD and SGD: Beyond Quadratic Potentials
Researchers have investigated the stability of nonlinear dynamics in gradient descent (GD) and stochastic gradient descent (SGD) optimization algorithms, moving beyond simplified quadratic potential assumptions. The study derives an exact criterion for stable oscillations in GD near minima, which depends on higher-order derivatives and generalizes existing findings. For SGD, the research indicates that nonlinear dynamics can diverge in expectation due to a single unstable batch, contrasting with linear analysis that suggests an average effect. The paper also proves that if all batches are linearly stable, the nonlinear dynamics of SGD remain stable in expectation. AI
IMPACT Provides a deeper theoretical understanding of optimization algorithms crucial for training large AI models.