A new research paper published on arXiv analyzes the update mechanism of the Muon model, identifying a flaw where it over-steps high-curvature spectral directions. The study, conducted using held-out gradients and Hessian-vector products, found that the model's multipliers increase monotonically from top singular directions to the tail. The researchers propose and test interventions, including orthogonalizing Q, K, and V matrices separately and applying a spectral tilt during the cooldown phase, which collectively lowered the loss of the model. AI
IMPACT Identifies specific areas for optimization in model update mechanisms, potentially leading to more efficient training and improved performance.
RANK_REASON Research paper published on arXiv detailing analysis and proposed improvements for a specific model. [lever_c_demoted from research: ic=1 ai=1.0]
- alphaXiv
- arXiv
- CatalyzeX
- Connected Papers
- DagsHub
- FineWeb-Edu
- Gotit.pub
- Hugging Face
- Lean
- Litmaps
- Muon
- NorMuon
- ScienceCast
- Scite
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →