Researchers have introduced DeMuon, a novel decentralized method for matrix optimization over graphs. This approach extends the centralized Muon algorithm by incorporating matrix orthogonalization through Newton-Schulz iterations and utilizing gradient tracking to handle local function heterogeneity. DeMuon achieves iteration complexity comparable to centralized algorithms, even under heavy-tailed noise, and is presented as the first direct extension of Muon to decentralized graph optimization with theoretical guarantees. Preliminary experiments show DeMuon outperforming other decentralized algorithms in transformer pretraining tasks across various network topologies. AI
IMPACT Introduces a new decentralized optimization method that could improve distributed AI training efficiency.
RANK_REASON The cluster contains a research paper detailing a new algorithm with theoretical guarantees and preliminary experimental results. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →