Researchers have identified a challenge in training Mixture-of-Experts (MoE) language models using the Muon optimizer, specifically concerning the balance of the first MoE layer. When Muon is used for the entire network, it can lead to significant routing imbalance in the initial layers, causing one expert to handle disproportionately more load. This issue can be addressed by either centering the router inputs or reducing Muon's learning rate for the optimizer. The study also found that keeping the router trained with Muon, rather than switching it to AdamW, is preferable at this scale. AI
IMPACT Identifies a specific training challenge for MoE models, offering solutions that could improve efficiency and stability.
RANK_REASON Academic paper detailing a novel finding about model training. [lever_c_demoted from research: ic=1 ai=1.0]
Read on Hugging Face Daily Papers →
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
