PulseAugur
中
实时 08:25:11
English(EN) Dense Local Dependencies Induce Attention-Logit Explosion and Training Instability During Long-Sequence Transformer Training

新研究探讨长上下文模型中的Transformer不稳定性

作者 PulseAugur 编辑部 · [2 个来源] ·

近期两篇arXiv论文深入探讨了长上下文Transformer的训练挑战。第一篇论文识别出一种称为秩崩溃（rank-collapse）的根本性病理，即随着上下文长度的增加，注意力分数变得均匀化，并提出使用$\\log n$的关键因子进行“注意力缩放”（attention scaling）以维持自适应注意力。第二篇论文将长序列Transformer的训练不稳定性，尤其是在低精度算术下，归因于密集局部依赖。该论文认为，这些依赖会导致“注意力-Logit爆炸”（attention-logit explosion），可以通过增加注意力维度或显式建模局部依赖来缓解。 AI

影响 这些论文为改进长上下文Transformer模型的稳定性和效率提供了理论见解，有望带来更强大的LLM。

排序理由 两篇发表在arXiv上的学术论文，详细介绍了Transformer架构局限性的理论和实证发现。

在 arXiv cs.LG 阅读 →

AI 生成摘要 · Google Gemini · 来自 2 个来源。 我们如何撰写摘要 →

新研究探讨长上下文模型中的Transformer不稳定性

本文如何被排名

Signal score
0 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Research
两篇发表在arXiv上的学术论文，详细介绍了Transformer架构局限性的理论和实证发现。
Source corroboration
2 independent sources
Multiple independent publishers reporting the same story raises confidence that it's real and newsworthy.
Topics
paper, infra
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
46 days old
Aged out of breaking-news scoring windows; ranking reflects the durable signal from the full source set.

完整方法见我们的编辑标准。

报道来源 [2]

  1. arXiv cs.LG TIER_1 English(EN) · Shi Chen, Zhengjiang Lin, Yury Polyanskiy, Philippe Rigollet ·

    长上下文Transformer中的关键注意力缩放

    arXiv:2510.05554v2 Announce Type: replace Abstract: As large language models scale to longer contexts, attention layers suffer from a fundamental pathology: attention scores collapse toward uniformity as context length $n$ increases, causing tokens to cluster excessively, a pheno…

  2. arXiv cs.LG TIER_1 English(EN) · Suvadeep Hajra ·

    密集局部依赖导致长序列Transformer训练期间的注意力-Logit爆炸和训练不稳定

    arXiv:2505.15548v2 Announce Type: replace Abstract: Autoregressive transformer language models frequently exhibit training instability when trained on long sequences, particularly under low-precision arithmetic. Although this instability is often accompanied by attention-logit ex…

相关实体

相关话题