Researchers have introduced DeltaMomentum, a novel approach to updating momentum in deep learning optimizers. Unlike traditional methods that use a fixed rate for exponential moving averages, DeltaMomentum dynamically adjusts the forgetting rate based on the frequency of gradient updates for specific directions. This key-value based method, inspired by the structure of linear layer gradients, aims to clear stale directions more efficiently and improve training speed. Experiments show that DeltaAdamW, an implementation of DeltaMomentum, achieves competitive validation losses in significantly fewer steps compared to standard AdamW across various model scales and datasets, including large language models and image classification tasks. AI
IMPACT Potentially accelerates training for large language models and other deep learning applications by improving gradient optimization efficiency.
RANK_REASON Academic paper introducing a novel method for deep learning optimizers. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
