English(EN)Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Experts
新研究探索专家混合模型的先进路由技术 · 跟踪 4 个来源
作者PulseAugur 编辑部·[6 个来源]·
研究人员正在探索新的方法来提高专家混合（MoE）模型的性能和专业化。一种方法侧重于对齐不同层之间路由状态的几何结构，以揭示共享的底层动态，从而提高预测能力。另一种方法，对比路由机制（CoRM），使用针对指数移动平均的对比方法来集中路由信号，从而实现更好的专家专业化和在推理基准上的准确性提升。第三篇论文为理解 MoE 架构提供了一个统计框架，分析了近似、专家学习和路由器估计之间的权衡，并解释了共享专家如何捕获常见的预测结构。
AI
arXiv:2609.04453v1 Announce Type: cross Abstract: Expert pruning reduces the memory and serving cost of Mixture-of-Experts (MoE) models by removing low-importance experts identified by the router, assuming router probabilities provide a reliable importance signal. We observe that…
arXiv:2609.04575v1 Announce Type: cross Abstract: Modern fine-grained Mixture-of-Experts (MoE) models route each token to a small number of experts and renormalize their router probabilities. We show that this renormalization implicitly calibrates expert output gain to the traini…
arXiv:2609.02404v1 Announce Type: cross Abstract: Sparse mixture-of-experts (MoE) models use an independently parameterized router at each sparse layer to select experts for every token. Prior work has shown that routing decisions across depth can often be predicted from earlier …
arXiv:2609.01100v1 Announce Type: new Abstract: In current Mixture-of-Experts architectures, routing is performed based on representations dominated by structure shared across all tokens, limiting expert specialization. We show that contrasting each token against an Exponential M…
In current Mixture-of-Experts architectures, routing is performed based on representations dominated by structure shared across all tokens, limiting expert specialization. We show that contrasting each token against an Exponential Moving Average of the layer's hidden states, rath…
arXiv:2609.03501v1 Announce Type: new Abstract: Mixture-of-experts (MoE) architectures increase model capacity by combining a collection of expert predictors through input-dependent routing, while often activating only a small subset of experts for each input. Despite their growi…
