Mixture of Experts (MoE) architecture, first proposed in 1991 by Jacobs et al., offers a solution to the scale vs. cost dilemma in large language models. Unlike dense models where all parameters are activated for every token, MoE layers use a gating network to selectively activate only a subset of specialized "expert" networks per token. This allows models like DeepSeek-V3, with hundreds of billions of parameters, to utilize a fraction of them for each computation, significantly reducing costs. The critical shift to sparse gating, introduced by Shazeer et al. in 2017, enabled actual compute savings by ensuring only the top-k selected experts process each token, a principle fundamental to modern MoE LLMs. AI
IMPACT Enables more efficient scaling of LLMs by reducing computational cost per token.
RANK_REASON The item discusses the historical development and technical details of the Mixture of Experts architecture in LLMs, referencing key papers and models. [lever_c_demoted from research: ic=1 ai=1.0]
- Adaptive Mixtures of Local Experts
- DeepSeek-V3
- Google Brain
- Hinton
- Jordan
- mixture of experts
- Nowlan
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
- Shazeer
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
