Researchers have investigated the dynamics of softmax routing within mixture-of-experts (MoE) models, observing that it can lead to a concentration of probability on a subset of experts, effectively suppressing others. The study explores the formation and persistence of this concentration, as well as the mechanisms for probability recovery. They identified a stationary-loss barrier for fixed experts and found that in models with trainable expert outputs, adaptation rates influence routing limits. The research also proposes a method to decompose covariance and gradient advantage, revealing logarithmic or power-law recovery times under specific conditions. Experiments with Transformer-MoE training demonstrated that altering router-gradient contrast impacts recovery and accumulated log-odds. AI
IMPACT This research provides insights into the internal dynamics of MoE models, potentially leading to improved training stability and expert utilization.
RANK_REASON The cluster contains a research paper detailing technical findings on model architecture. [lever_c_demoted from research: ic=1 ai=1.0]
Read on Hugging Face Daily Papers →
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
