A new research paper proposes the Affine Inverse-square-Root decay (AIR) learning rate schedule, which is designed to optimize training for models using weight-normalized (WN) techniques. The study, led by Qifan Yu, found that the effective learning rate trajectory, rather than the nominal learning rate, is crucial for performance differences between WN and conventional weight decay. AIR aims to match specific peak and terminal learning rates, and experiments on large Mixture-of-Experts models showed it outperformed standard schedules in reducing training loss and improving downstream performance. The research suggests that these gains are primarily due to the scheduling of the effective learning rate itself. AI
IMPACT Optimizes training for large AI models, potentially leading to faster convergence and improved performance.
RANK_REASON The cluster contains a research paper detailing a new method for optimizing AI model training. [lever_c_demoted from research: ic=1 ai=1.0]
- AIR (Affine Inverse-square-Root decay)
- alphaXiv
- arXiv
- DagsHub
- Hugging Face
- mixture of experts
- Qifan Yu
- Weight Decay (WD)
- Weight-Normalized (WN)
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →