PulseAugur
中
实时 07:48:58
English(EN) Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shuffling

新的Spectral-MoE方法优化了LLM跨层的专家利用率

研究人员开发了Spectral-MoE,一种将密集大型语言模型转换为混合专家(MoE)架构的新颖方法。与之前在所有层使用统一专家利用率的方法不同,Spectral-MoE允许异构专家分配,这意味着不同层可以使用不同数量的专家。这种方法在Llama-2 7B和Llama-3 8B等模型上得到了验证,与统一利用率相比,它显著降低了困惑度并提高了任务准确性。另一项相关技术侧重于通过利用相关性专家分配来减少MoE预训练中的通信开销,这可以大大缩短训练时间。 AI

影响 这些技术为降低训练和部署大型语言模型的计算和通信成本提供了巨大潜力。

排序理由 该集群包含两篇研究论文,详细介绍了优化混合专家(MoE)模型的新颖方法。

在 Hugging Face Daily Papers 阅读 →

AI 生成摘要 · Google Gemini · 来自 2 个来源。 我们如何撰写摘要 →

新的Spectral-MoE方法优化了LLM跨层的专家利用率

本文如何被排名

Signal score
0 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Research
该集群包含两篇研究论文,详细介绍了优化混合专家(MoE)模型的新颖方法。
Source corroboration
2 independent sources
Multiple independent publishers reporting the same story raises confidence that it's real and newsworthy.
Topics
paper, infra
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
46 days old
Aged out of breaking-news scoring windows; ranking reflects the durable signal from the full source set.

完整方法见我们的编辑标准。

报道来源 [2]

  1. arXiv cs.LG TIER_1 English(EN) · Zachary Broveak, Christian Kubicka, Thierry Tambe ·

    Spectral-MoE:密集到MoE转换中的异构逐层专家利用

    arXiv:2610.19232v1 Announce Type: new Abstract: Large language models impose substantial memory demands and arithmetic costs during inference. Dense-to-mixture-of-experts (MoE) conversion is a model optimization technique that aims to reduce these arithmetic costs. Dense-to-MoE c…

  2. Hugging Face Daily Papers TIER_1 English(EN) ·

    MoE预训练中的专家耦合:通过相关性放置和Token打乱减少全对全开销

    Mixture-of-Experts (MoE) layers replace the feed-forward block of a Transformer with E expert networks, and each token is routed to k of these experts. Under expert parallelism (EP) the experts are distributed across GPUs, and every MoE layer runs all-to-all collectives in the fo…