A comparison of open Mixture-of-Experts (MoE) models reveals significant divergence in their routing mechanisms, despite a shared core idea. Models vary widely in expert count, ranging from 8 to 384, and the number of experts selected per token, from 1 to 8. Some models incorporate a continuously active shared expert, while others omit this feature entirely. The choice of routing strategy impacts memory requirements and computational efficiency, with some models like DeepSeek-V4-Pro requiring multi-GPU setups and others, such as gpt-oss-120b, fitting on a single GPU. AI
IMPACT Understanding MoE routing differences is crucial for optimizing performance and resource allocation in deploying large language models.
RANK_REASON Comparison of technical details of multiple open-source MoE models. [lever_c_demoted from research: ic=1 ai=1.0]
- DeepSeek
- DeepSeek-V3
- DeepSeek V4
- DeepSeek V4-Pro
- GLM-4.5
- gpt-oss
- GPT-OSS 120B
- Kimi K2
- Llama 4
- Llama 4 Maverick
- Mixtral
- MXFP4
- Qwen
- Qwen2.5-MoE
- Qwen3
- Qwen3.6 35B-A3B
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
