A new research paper explores how different optimizers perform as training horizons and parameter counts increase. The study found that the optimal hyperparameters and relative performance of optimizers like Muon, SOAP, ADANA, and AdamW change significantly with extended training. Specifically, the preferred learning rate schedule can reverse, and weight decay coefficients scale with the square root of the overtraining factor. ADANA showed a persistent scaling advantage over AdamW, especially with log-time weight decay and momentum cooldown, becoming competitive with SOAP at higher overtraining levels. Muon and SOAP offered consistent token-efficiency advantages over AdamW across most tested ranges. AI
IMPACT Findings suggest training horizon is a critical factor in optimizer selection and design for large-scale models.
RANK_REASON Research paper published on arXiv detailing optimizer performance. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
