Brief

Researchers have identified a key mechanism behind the loss spikes that frequently occur during neural network training with the Adam optimizer. Their analysis reveals that these spikes are not solely due to landscape geometry but stem from the internal dynamics of Adam's second moment estimator. Specifically, a decoupling between the adaptive preconditioner and instantaneous squared gradients causes the preconditioner to decay autonomously, leading to instability and dramatic loss increases. AI

Researchers have theoretically analyzed the Adam optimization algorithm, identifying a specific class of highly degenerate polynomials where it converges automatically without external schedulers. This work demonstrates that Adam achieves local linear convergence on these functions, outperforming Gradient Descent and Momentum due to an exponential amplification of the effective learning rate. The study also characterizes Adam's hyperparameter phase diagram, revealing three distinct behavioral regimes: stable convergence, spikes, and SignGD-like oscillation. AI