Perfect Parallelization in Mini-Batch SGD with Classical Momentum Acceleration
Researchers have developed a new theory explaining how classical momentum schemes like Polyak's heavy ball can accelerate stochastic gradient descent (SGD) for large-scale machine learning. The theory applies to quadratics in the interpolation regime and accommodates arbitrary mini-batch sizes with minimal noise assumptions. A key finding is that momentum-driven acceleration scales directly with the gradient mini-batch size, enabling perfect parallelization of computations. AI
IMPACT This theoretical advance could lead to more efficient training of large-scale machine learning models by enabling better parallelization of computations.