Sonar-MoE 将 MoE 训练内存减少 22%，并加快了过程

作者 PulseAugur 编辑部 · [1 个来源] · 2026-11-04 05:00

研究人员开发了 Sonar-MoE，一种用于训练混合专家（MoE）模型的新方法，可显著减少内存使用量。通过在反向传播过程中更新专家梯度并立即释放它们，Sonar-MoE 确保 GPU 一次只持有专家梯度的一层。与标准方法相比，该技术已显示出将峰值内存消耗降低 19-22%，并将训练时间缩短 17-23%，同时保持相同的更新和可比的留出损失。 AI

影响这项技术可以使在现有硬件上训练更大、更复杂的混合专家模型成为可能，从而可能加速该领域的研究和开发。

排序理由详细介绍 MoE 模型新颖训练技术的学术论文。[lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.CL 阅读 →

AI 生成摘要 · Google Gemini · 来自 1 个来源。我们如何撰写摘要 →

本文如何被排名

Signal score: 41 / 100; Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket: Tool; 详细介绍 MoE 模型新颖训练技术的学术论文。[lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration: Single-source cluster; Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics: paper, infra; Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance: High; Clearly on-topic for AI-industry coverage.
Story freshness: Breaking (< 6h); Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

完整方法见我们的编辑标准。

报道来源 [1]

arXiv cs.CL TIER_1 English(EN) · Avni Mittal, Avinash Anand, Ashutosh Kumar, Dikshant Kukreja, Kritarth Prasad, Chaitanya Lakhchaura, Rishitej Reddy Vyalla, Erik Cambria, Zhengkui Wang, Rajiv Ratn Shah · 2026-11-04 05:00

Sonar-MoE：用于内存高效混合专家训练的向后专家更新

arXiv:2611.03577v1 Announce Type: cross Abstract: Most parameters of a Mixture-of-Experts (MoE) model belong to its experts, and standard training keeps every expert gradient in GPU memory until the optimizer step. This memory grows with the number and size of the experts, so it …

本文如何被排名

报道来源 [1]

Sonar-MoE：用于内存高效混合专家训练的向后专家更新

相关实体

相关话题