English(EN) Hyperparameter Scaling Laws Across MoE Sparsity

新的缩放定律为超稀疏MoE模型解锁超参数调优

作者 PulseAugur 编辑部 · [1 个来源] · 2026-09-10 04:00

研究人员开发了专门针对超稀疏混合专家（MoE）模型的新超参数缩放定律，解决了在不同稀疏度级别之间转移最优学习率和批量大小的挑战。通过涉及1800个实验和约20万亿个标记的大量预训练运行，他们确定了两种不同的缩放机制。研究结果表明，最优批量大小随训练标记而缩放，而学习率受训练计算的影响，并且对模型大小和数据分配具有鲁棒性。这些新定律将激活率作为一个乘法因子，能够更准确地预测稀疏MoE模型的超参数，即使对于拥有数十亿参数和极低专家激活的模型也是如此。 AI

影响通过提供准确的超参数指导，实现稀疏MoE模型更高效的训练和部署。

排序理由详细介绍MoE模型新缩放定律的学术论文。[lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.AI 阅读 →

AI 生成摘要 · Google Gemini · 来自 1 个来源。我们如何撰写摘要 →

本文如何被排名

Signal score: 12 / 100; Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket: Tool; 详细介绍MoE模型新缩放定律的学术论文。[lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration: Single-source cluster; Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics: paper, model release; Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance: High; Clearly on-topic for AI-industry coverage.
Story freshness: Breaking (< 6h); Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

完整方法见我们的编辑标准。

报道来源 [1]

arXiv cs.AI TIER_1 English(EN) · Changxin Tian, Kunlong Chen, Jia Liu, Ziqi Liu, Zhiqiang Zhang, Jun Zhou · 2026-09-10 04:00

MoE 稀疏性下的超参数缩放定律

arXiv:2609.08690v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models expand model capacity without a proportional increase in training compute, but increasing sparsity makes reliable hyperparameter transfer challenging. In this work, we show that conventional hyperpa…

本文如何被排名

报道来源 [1]

MoE 稀疏性下的超参数缩放定律

相关实体

相关话题