Researchers have introduced iMuon, a novel optimization framework designed for manifold-valued parameters in machine learning. This approach addresses limitations of existing optimizers like Muon by preserving symmetries and coupling constraints on various matrix manifolds, including fixed-rank, SPD, Stiefel, and Grassmann. iMuon offers closed-form updates and establishes convergence rates that are independent of manifold dimension and conditioning, demonstrating effectiveness in experiments with LoRA fine-tuning of LLMs, image classification, and subspace learning. AI
IMPACT Introduces a novel optimization method that could improve efficiency and applicability of LLM fine-tuning and other manifold-based learning tasks.
RANK_REASON The cluster contains a research paper detailing a new optimization framework for machine learning. [lever_c_demoted from research: ic=1 ai=1.0]
- arXiv
- Hugging Face
- iMuon
- LoRA+
- Muon
- Nanyang Technological University
- Riemannian Matrix Manifolds
- Yibang Li
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →