PulseAugur
中
实时 11:10:11

Zyphra AI 发布用于 LLM 的亚二次方注意力方法

作者 PulseAugur 编辑部 · [4 个来源] ·

Zyphra AI 推出了 Zyphra Hierarchical Attention (ZHA)，这是一种新颖的稀疏注意力机制，旨在克服大型语言模型中传统注意力的二次方缩放限制。ZHA 采用树状索引器，可递归压缩键摘要，使查询能够高效检索相关信息，而无需扫描整个上下文。即使通过相对少量的数据将其蒸馏到现有的预训练模型中，该方法在长上下文任务（包括推理和检索）上仍能保持性能。虽然尚未提供优化内核，但 ZHA 在将有效上下文窗口扩展到数亿甚至数十亿个 token 方面显示出巨大潜力。 AI

影响 该方法可以显著扩展 LLM 的实际上下文窗口，从而实现更复杂的推理和数据处理。

排序理由 该集群描述了一篇关于改进 LLM 注意力机制的新颖方法的最新研究论文。

在 arXiv cs.AI 阅读 →

AI 生成摘要 · Google Gemini · 来自 4 个来源。 我们如何撰写摘要 →

Zyphra AI 发布用于 LLM 的亚二次方注意力方法

本文如何被排名

Signal score
3 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Research
该集群描述了一篇关于改进 LLM 注意力机制的新颖方法的最新研究论文。
Source corroboration
4 independent sources
Strong cross-source corroboration — multiple independent publishers covered this within the clustering window.
Topics
model release, infra
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
1 days old
Coverage has settled into its steady-state source set.
Coverage growth since scoring
+1 source(s) since last score
New sources have picked up this story since our last re-score. Score will update on the next scoring pass.

完整方法见我们的编辑标准。

报道来源 [4]

  1. arXiv cs.CL TIER_1 English(EN) · Leon Lufkin, Cutter Dawes, Alexander Speicher, Kamesh Krishnamurthy, Tom\'as Figliolia, Beren Millidge ·

    Subquadratic Attention with a Hierarchical Tree-Structured Indexer

    arXiv:2610.42112v1 Announce Type: new Abstract: The quadratic scaling of attention is a fundamental bottleneck limiting the long-context capabilities of large language models (LLMs). A variety of approaches have attempted to circumvent this, but they either fail to match attentio…

  2. arXiv cs.AI TIER_1 English(EN) · Ales Kubicek, Andrea Caciolai, Liangzu Peng, Sukrit Sumant, Benjamin Hoover, Fraser Greenlee, Ruairidh M. Battleday, Radim Urban, Torsten Hoefler, James C. R. Whittington ·

    Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention

    arXiv:2610.42147v1 Announce Type: new Abstract: Agentic workflows are increasingly shifting towards long-horizon tasks, such as open-ended scientific work, where models benefit from long context windows. However, at ultra-long context (above 1M tokens), full attention is prohibit…

  3. MarkTechPost TIER_1 English(EN) · Asif Razzaq ·

    Zyphra AI Introduces Zyphra Hierarchical Attention (ZHA): A Path to True Subquadratic Attention

    <p>Zyphra Hierarchical Attention (ZHA) is a tree-indexed sparse attention method that retrofits frozen LLMs with 4B tokens of distillation, matching dense attention on reasoning and 256K retrieval benchmarks.</p> <p>The post <a href="https://www.marktechpost.com/2026/11/02/zyphra…

  4. Mastodon — mastodon.social TIER_1 English(EN) · [email protected] ·

    Zyphra's new sparse attention method searches a tree of summarised keys instead of scanning the whole context, where indexer cost dominates near 1M tokens. Dist

    Zyphra's new sparse attention method searches a tree of summarised keys instead of scanning the whole context, where indexer cost dominates near 1M tokens. Distilled into frozen models with 4B tokens, it adds 81,920 trainable parameters to Qwen3.6-35B-A3B and stays close to dense…

    链接 marktechpost.com/…/zyphra-hierarchical-at…

相关实体

相关话题