滑动窗口注意力在大型语言模型中优于线性注意力

报道来源 [5]

1. arXiv cs.CL TIER_1 English(EN) · Alexia Jolicoeur-Martineau, Rhea Sanjay Sukthanker, Pashmina Cameron, Emy Gervais · 2026-08-31 04:00

   滑动窗口优于线性注意力

   arXiv:2608.28444v1 Announce Type: new Abstract: Due to the nature of quadratic attention, Large Language Models (LLMs) consume a lot of memory and energy. Every new token costs more than the previous one. For each additional token, the keys and values must be stored in memory ind…

2. Hugging Face Daily Papers TIER_1 English(EN) · 2026-08-28 00:00

   滑动窗口超越线性注意力

   Sliding window attention with sinks outperforms post-trained linear attention on long-context tasks without requiring retraining, offering a cheaper and more reliable inference solution.

3. dev.to — LLM tag TIER_1 English(EN) · Prabhakar Chaudhary · 2026-08-31 21:32

   技术分析：滑动窗口在效率基准测试中优于线性注意力

   <h1> Why Sliding-Window Attention Outperforms Linear Attention for Efficient Inference </h1> <p>The pursuit of infinite context windows in Large Language Models (LLMs) has led to a proliferation of complex architectural modifications. As models scale, the standard quadratic atten…

4. r/MachineLearning TIER_1 English(EN) · /u/Justgototheeffinmoon · 2026-08-31 16:35

   滑动窗口注意力机制在长上下文推理方面优于线性注意力机制 [R]

   <!-- SC_OFF --><div class="md"><p>Sliding Window Attention with sinks, one of the simplest existing fixes for the quadratic-cost problem in LLMs, holds up as well or better than the linear-attention variants labs have been spending post-training compute to produce. That is the cl…

5. r/LocalLLaMA TIER_1 English(EN) · /u/woadwarrior · 2026-08-31 14:12

   滑动窗口超越线性注意力

   <!-- SC_OFF --><div class="md"><p>Interesting new <a href="https://arxiv.org/abs/2608.28444">paper</a> from Alexia Jolicoeur-Martineau (of Tiny Recursive Model fame) and collaborators. They seem to be able to replace quadratic attention with sliding window attention + attention s…