Rethinking Sparse Mixture of Experts from a Unified Perspective
Two new research papers propose novel frameworks for optimizing Sparse Mixture of Experts (SMoE) models. The first, Unified Sparse Mixture of Experts (USMoE), reframes SMoE through linear programming to create a unified mechanism and score, improving performance across various tasks and data types. The second, Nash Merging of Experts (NAMEx), applies game theory and Nash Bargaining to expert merging, enhancing collaboration and efficiency. NAMEx has demonstrated effectiveness on large-scale models like Qwen1.5-MoE and DeepSeek-MoE. AI
IMPACT These advancements in SMoE architectures could lead to more efficient and powerful AI models across various domains.