English(EN) SpectralShift: Effective Context Window Extension of Gated DeltaNet via Spectral Reparameterization

SpectralShift通过谱重参数化增强GDN模型的上下文窗口

作者 PulseAugur 编辑部 · [1 个来源] · 2026-09-15 04:00

研究人员推出了一种新颖的方法SpectralShift，用于扩展门控DeltaNet (GDN) 模型的上下文窗口，该模型利用线性注意力层。该方法侧重于转换矩阵的光谱特性，发现宽慢光谱带和快衰减模式的保留对于有效的长距离信息检索和上下文切换至关重要。SpectralShift对alpha投影进行重参数化以增强慢传播能力，并采用学习率缩放来促进alpha投影的训练。实验表明，SpectralShift显著提高了GDN的长上下文能力，为扩展其上下文窗口提供了一个有效的解决方案。 AI

影响增强了线性注意力模型的长上下文能力，可能提高需要广泛信息回忆的任务的性能。

排序理由该集群包含一篇学术论文，详细介绍了一种扩展语言模型上下文窗口的新方法。 [lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.CL 阅读 →

AI 生成摘要 · Google Gemini · 来自 1 个来源。我们如何撰写摘要 →

本文如何被排名

Signal score: 18 / 100; Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket: Tool; 该集群包含一篇学术论文，详细介绍了一种扩展语言模型上下文窗口的新方法。 [lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration: Single-source cluster; Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics: paper, infra; Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance: High; Clearly on-topic for AI-industry coverage.
Story freshness: Breaking (< 6h); Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

完整方法见我们的编辑标准。

报道来源 [1]

arXiv cs.CL TIER_1 English(EN) · Zian Liu, Yiwen Hu, Zican Dong, Tian Xie, Wayne Xin Zhao, Yucheng Ding, Ran Tao, Bryan Dai · 2026-09-15 04:00

SpectralShift：通过谱重参数化实现门控DeltaNet的有效上下文窗口扩展

arXiv:2609.14320v1 Announce Type: new Abstract: Recently, linear attention layers have been increasingly adopted to replace softmax attention at scale for long-context modeling. However, existing context extension approaches typically apply continued pretraining directly without …

本文如何被排名

报道来源 [1]

SpectralShift：通过谱重参数化实现门控DeltaNet的有效上下文窗口扩展

相关实体

相关话题