PulseAugur
中
实时 15:25:18
English(EN) SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking

新方法提升LLM稀疏注意力效率

研究人员开发了两种新颖的方法来提高大型语言模型中稀疏注意力机制的效率。第一种,HISA(分层索引稀疏注意力),引入了一个两阶段的索引过程,首先过滤令牌块,然后在这些块内进行精炼选择,在无需重新训练的情况下实现显著的加速。第二种,SAS(简单注意力稀疏化),使用一个端到端与语言建模损失一起训练的连续门控机制来优化上下文排名,其性能优于现有的可训练稀疏注意力方法,尤其是在注意力预算紧张的情况下。 AI

影响 稀疏注意力方面的这些进步可以显著降低计算成本,并使LLM能够更有效地处理更长的上下文。

排序理由 该集群包含两篇研究论文,介绍了用于改进LLM中稀疏注意力机制的新颖方法。

在 Hugging Face Daily Papers 阅读 →

AI 生成摘要 · Google Gemini · 来自 3 个来源。 我们如何撰写摘要 →

新方法提升LLM稀疏注意力效率

本文如何被排名

Signal score
0 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Research
该集群包含两篇研究论文,介绍了用于改进LLM中稀疏注意力机制的新颖方法。
Source corroboration
3 independent sources
Multiple independent publishers reporting the same story raises confidence that it's real and newsworthy.
Topics
paper, infra
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
72 days old
Aged out of breaking-news scoring windows; ranking reflects the durable signal from the full source set.
Coverage growth since scoring
+1 source(s) since last score
New sources have picked up this story since our last re-score. Score will update on the next scoring pass.

完整方法见我们的编辑标准。

报道来源 [3]

  1. arXiv cs.CL TIER_1 English(EN) · Zhiwei Li, Lei Zhu, Hao Gu, Xiang Hu, Yan Wang, Haitao Mi, Sirui Han, Leo Liang, Zhijiang Guo ·

    SAS:通过端到端上下文排序优化实现简单注意力稀疏化

    arXiv:2609.13141v1 Announce Type: new Abstract: Post-training attention sparsification reduces the quadratic cumulative attention cost of pretrained Transformers by selecting a small set of context units (tokens or blocks) for each query. Existing trainable methods usually use a …

  2. arXiv cs.LG TIER_1 English(EN) · Yufei Xu, Fanxu Meng, Fan Jiang, Yuxuan Wang, Ruijie Zhou, Zhaohui Wang, Jiexi Wu, Zhixin Pan, Xiaojuan Tang, Wenjie Pei, Tongxuan Liu, Di Yin, Xing Sun, Muhan Zhang ·

    HISA:高效分层索引实现细粒度稀疏注意力

    arXiv:2603.28458v4 Announce Type: replace Abstract: Token-level sparse attention mechanisms, exemplified by DeepSeek Sparse Attention (DSA), achieve fine-grained key selection by scoring every historical key for each query through a lightweight indexer, then computing attention o…

  3. Hugging Face Daily Papers TIER_1 English(EN) ·

    SAS:通过端到端上下文排名优化实现简单注意力稀疏化

    SAS improves sparse attention by training a selector end-to-end with language modeling loss via continuous gating inside attention softmax, yielding better context ranking under tight budgets.