DeltaAdamW

New DeltaMomentum optimizer speeds up deep learning training

Researchers have introduced DeltaMomentum, a novel approach to updating momentum in deep learning optimizers. Unlike traditional methods that use a fixed rate for exponential moving averages, DeltaMomentum dynamically a…