PulseAugur
中
实时 06:31:43
English(EN) SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers

SMELT 架构通过循环中间层提高 Transformer 效率

作者 PulseAugur 编辑部 · [2 个来源] ·

研究人员开发了一种名为 SMELT（稀疏 MoE Transformer，中间层循环两次）的新架构，该架构通过迭代共享层块来改进循环 Transformer。通过精确匹配 FLOPs、参数和 KV 缓存，SMELT 在计算方面展示了更快的损失下降，可能节省 6.8-18.0% 的训练 FLOPs。这种架构优势转化为下游基准测试（尤其是在编码任务中）的性能提升，并且似乎源于中间层的第二次通过减少了注意力沉淀并重新将焦点导向相关 token。 AI

影响 这项研究提供了一种提高 Transformer 效率的实用方法，可能降低训练成本并提高在编码等特定任务上的性能。

排序理由 该集群包含一篇详细介绍新模型架构及其性能特征的研究论文。[lever_c_demoted from research: ic=1 ai=1.0]

在 Hugging Face Daily Papers 阅读 →

AI 生成摘要 · Google Gemini · 来自 2 个来源。 我们如何撰写摘要 →

SMELT 架构通过循环中间层提高 Transformer 效率

本文如何被排名

Signal score
1 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Tool
该集群包含一篇详细介绍新模型架构及其性能特征的研究论文。[lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration
2 independent sources
Multiple independent publishers reporting the same story raises confidence that it's real and newsworthy.
Topics
paper, model release
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
1 days old
Coverage has settled into its steady-state source set.
Coverage growth since scoring
+1 source(s) since last score
New sources have picked up this story since our last re-score. Score will update on the next scoring pass.

完整方法见我们的编辑标准。

报道来源 [2]

  1. arXiv cs.LG TIER_1 English(EN) · Shaowen Wang, Ge Zhang, Kairong Luo, Yuhao Wu, Shaofan Liu, Jiaheng Liu, Wenhao Huang, Shen Yan, Jian Li ·

    SMELT：面向计算匹配的MoE循环Transformer的扩展定律

    arXiv:2609.01343v1 Announce Type: new Abstract: Looped Transformers increase effective depth by iterating a shared block of layers, but most evaluations compare at fixed model size, conflating architectural advantage with extra FLOPs. We study looping on Mixture-of-Experts Transf…

  2. Hugging Face Daily Papers TIER_1 English(EN) ·

    SMELT: 规模化定律用于计算匹配的MoE循环Transformer

    Looping middle layers in sparse Mixture-of-Experts Transformers improves training efficiency and downstream performance while matching per-token FLOPs, parameters, and cache budgets.

相关实体

相关话题