Nesterov-style momentum

New theory shows momentum enables perfect parallelization in SGD

Researchers have developed a new theory explaining how classical momentum schemes like Polyak's heavy ball can accelerate stochastic gradient descent (SGD) for large-scale machine learning. The theory applies to quadrat…