PulseAugur
EN
LIVE 06:34:58

SMELT architecture improves Transformer efficiency by looping middle layers

By PulseAugur Editorial · [2 sources] ·

Researchers have developed a new architecture called SMELT (Sparse MoE Transformer, middle layers Loop Twice) which improves upon Looped Transformers by iterating on a shared block of layers. By closely matching FLOPs, parameters, and KV cache, SMELT demonstrates a faster loss drop with compute, potentially saving 6.8-18.0% of training FLOPs. This architectural advantage translates to improved performance on downstream benchmarks, particularly in coding tasks, and appears to stem from the second pass through middle layers reducing attention sinks and redirecting focus to relevant tokens. AI

IMPACT This research offers a practical recipe for improving Transformer efficiency, potentially reducing training costs and enhancing performance on specific tasks like coding.

RANK_REASON The cluster contains a research paper detailing a new model architecture and its performance characteristics. [lever_c_demoted from research: ic=1 ai=1.0]

Read on Hugging Face Daily Papers →

AI-generated summary · Google Gemini · from 2 sources. How we write summaries →

SMELT architecture improves Transformer efficiency by looping middle layers

How we ranked this

Signal score
1 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Tool
The cluster contains a research paper detailing a new model architecture and its performance characteristics. [lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration
2 independent sources
Multiple independent publishers reporting the same story raises confidence that it's real and newsworthy.
Topics
paper, model release
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
1 days old
Coverage has settled into its steady-state source set.
Coverage growth since scoring
+1 source(s) since last score
New sources have picked up this story since our last re-score. Score will update on the next scoring pass.

Full methodology in our editorial standards.

COVERAGE [2]

  1. arXiv cs.LG TIER_1 English(EN) · Shaowen Wang, Ge Zhang, Kairong Luo, Yuhao Wu, Shaofan Liu, Jiaheng Liu, Wenhao Huang, Shen Yan, Jian Li ·

    SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers

    arXiv:2609.01343v1 Announce Type: new Abstract: Looped Transformers increase effective depth by iterating a shared block of layers, but most evaluations compare at fixed model size, conflating architectural advantage with extra FLOPs. We study looping on Mixture-of-Experts Transf…

  2. Hugging Face Daily Papers TIER_1 English(EN) ·

    SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers

    Looping middle layers in sparse Mixture-of-Experts Transformers improves training efficiency and downstream performance while matching per-token FLOPs, parameters, and cache budgets.

RELATED ENTITIES

RELATED TOPICS