Researchers have developed a new optimization method called Quadratic Spectral Descent (QSD) that improves upon the existing Muon algorithm for training large language models. QSD incorporates local curvature information into the optimization process, which can alter both the singular values and directions of the optimal update. By approximating curvature with Kronecker-factored statistics and using a Frank-Wolfe approach, QSD is made practical. Experiments on GPT pre-training demonstrate that QSD consistently lowers validation loss compared to Muon and its variants, and can reduce training time by up to 8.49% while achieving matched validation loss. AI
IMPACT This new optimization technique could lead to more efficient training of large language models, potentially reducing computational costs and training times.
RANK_REASON The cluster contains a research paper detailing a new optimization method for LLMs. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
