New scaled weight decay method accelerates neural network training

Researchers have introduced a novel method for scaling weight decay, inspired by the Robbins-Monro conditions, to improve neural network training. This technique adjusts weight decay based on the fraction of the peak learning rate, ensuring asymptotic stationarity and avoiding the bias introduced by constant weight decay. When applied to mixture-of-experts models using the Muon optimizer, this scaled weight decay (Muon-SW) demonstrated a significant speedup, achieving target validation loss up to 30% faster at large scales. AI

IMPACT Promises to substantially accelerate the pre-training of frontier models with minimal implementation effort.

RANK_REASON The cluster contains a research paper detailing a new method for training neural networks. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• alphaXiv
• arXiv
• CatalyzeX
• DagsHub
• Gotit.pub
• Hugging Face
• IArxiv
• Muon
• Muon-SW
• Robbins--Monro
• ScienceCast

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →