Nesterov momentum stability analyzed for large neural networks

Researchers have analyzed the stability of Nesterov momentum when applied to stochastic pseudo-gradients in large neural networks trained across multiple workers. They derived a mean-square stability criterion for random quadratics, indicating that the inner batch size dictates whether more local steps lead to stability or instability. The study also found that while averaging pseudo-gradients can allow for higher outer momentum, the number of workers needed for stability might increase exponentially with local steps. These dynamics can be probed using shared-minibatch perturbations at intermediate neural network checkpoints. AI

IMPACT Provides theoretical insights into optimizing large-scale neural networks, potentially improving training efficiency and stability.

RANK_REASON Academic paper detailing theoretical analysis and experimental findings on optimization stability in distributed training. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• Hugging Face
• Nesterov momentum
• Neural Networks
• pseudo-gradients

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →