研究人员正在探索优化稀疏专家混合（MoE）模型的新方法，超越传统手段。一项研究引入了MOSAIC框架，该框架整合了架构和系统协同设计，通过考虑硬件约束和算法选择来提高模型效率和性能。另一篇论文研究了解耦MoE路由器中专家调度（选择）和聚合（加权）的角色，提出了一种可以独立优化的后计算头，以增强语言建模目标。 AI
影响 这些研究为优化MoE模型提供了新途径，有望带来更高效、更强大的大型语言模型。
排序理由 两篇学术论文提出了优化稀疏专家混合模型的新方法。
在 Hugging Face Daily Papers 阅读 →
AI 生成摘要 · Google Gemini · 来自 3 个来源。 我们如何撰写摘要 →
完整方法见我们的编辑标准。
arXiv:2608.10605v1 Announce Type: cross Abstract: In large-scale pretraining, the algorithm, architecture, and systems decisions are conventionally made in disconnected stages. A scaling law stage selects an architecture and training recipe, optimizing loss under compute constrai…
In large-scale pretraining, the algorithm, architecture, and systems decisions are conventionally made in disconnected stages. A scaling law stage selects an architecture and training recipe, optimizing loss under compute constraints, and a separate systems stage then optimizes t…
arXiv:2608.08853v1 Announce Type: new Abstract: Sparse Mixture-of-Experts (MoE) routers commonly use the same scores both to select experts and to weight their already-computed outputs. We study whether these two roles, dispatch and aggregation, should be coupled. On pretrained O…