Researchers have developed a minimal dynamical model to understand load imbalance in adaptive softmax routing for Mixture-of-Experts (MoE) layers. This model, derived from a reinforcement learning rule, exhibits a pitchfork bifurcation where a stable balanced state transitions to two asymmetric states above a critical feedback strength. Further analysis reveals a cusp catastrophe in the control-parameter plane when external asymmetry is introduced, with exact parametric equations provided for this phenomenon. The findings are supported by numerical experiments on a small MoE model and classification tasks, offering a low-dimensional mechanism for abrupt load imbalance in MoE routers. AI
IMPACT Provides a theoretical framework for understanding and potentially mitigating load imbalance in MoE architectures.
RANK_REASON Academic paper detailing a new model for understanding MoE router behavior. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
