PulseAugur
中
实时 18:52:56
English(EN) Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval

语言模型因注意力竞争而非长度而难以处理长输入

作者 PulseAugur 编辑部 · [2 个来源] ·

一项新研究表明，语言模型在处理长输入时遇到困难，并非由于序列长度本身，而是由于模型内部的注意力“竞争”。研究人员发现，对于 Qwen2.5-1.5B 等模型，中性填充文本对高达 32K 标记的准确性影响很小。然而，引入竞争性陈述或证据格式的文本会显著降低性能，注意力质量与准确性损失密切相关。这表明检索任务中的上下文衰减主要是注意力竞争问题，而不是简单的长度限制。 AI

影响 这项研究表明，提高长上下文性能可能需要架构更改以更好地管理注意力竞争，而不是仅仅增加上下文窗口大小。

排序理由 研究论文，详细介绍了语言模型行为的发现。

在 Hugging Face Daily Papers 阅读 →

AI 生成摘要 · Google Gemini · 来自 2 个来源。 我们如何撰写摘要 →

语言模型因注意力竞争而非长度而难以处理长输入

本文如何被排名

Signal score
0 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Research
研究论文，详细介绍了语言模型行为的发现。
Source corroboration
2 independent sources
Multiple independent publishers reporting the same story raises confidence that it's real and newsworthy.
Topics
paper, model release
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
7 days old
Aged out of breaking-news scoring windows; ranking reflects the durable signal from the full source set.

完整方法见我们的编辑标准。

报道来源 [2]

  1. Hugging Face Daily Papers TIER_1 English(EN) ·

    长度廉价，竞争昂贵：注意力预测检索中的上下文衰减

    Why do language models fail on long inputs? The common answer is length. We show that, for retrieval, length is cheap and competition is not. On five open models (0.5B to 3B parameters, four families), neutral filler costs nothing: accuracy stays at 0.97 to 1.00, up to 32K tokens…

  2. Hugging Face Daily Papers TIER_1 English(EN) ·

    长度不值钱，竞争才值钱：注意力预测检索中的上下文衰减

    In retrieval tasks, language model failure on long inputs stems from attention competition rather than sequence length, as rival evidence statements sharply reduce accuracy while neutral filler does not.

相关实体

相关话题