Transformer注意力中的谱异常值揭示了主导的学习结构

作者 PulseAugur 编辑部 · [1 个来源] · 2026-08-11 04:00

研究人员应用Marchenko-Pastur随机矩阵理论分析了预训练Transformer的注意力权重，识别出代表主导学习结构的谱异常值。通过将Mistral-7B模型中识别出的这些异常值归零，模型在HellaSwag和MMLU等基准测试上的性能显著下降，接近随机猜测水平。对多个Transformer的分析揭示了重复出现的模式，例如Q投影包含最多的异常值，以及特定的残差流维度在各层中形成带状异常值，这表明可能在参数高效微调和结构化剪枝方面有应用潜力。 AI

影响这项研究可能带来更高效的大型语言模型微调和剪枝方法。

排序理由学术论文，详细介绍了Transformer注意力权重的新分析方法。[lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.AI 阅读 →

AI 生成摘要 · Google Gemini · 来自 1 个来源。我们如何撰写摘要 →

本文如何被排名

Signal score: 0 / 100; Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket: Tool; 学术论文，详细介绍了Transformer注意力权重的新分析方法。[lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration: Single-source cluster; Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics: paper, model release; Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance: High; Clearly on-topic for AI-industry coverage.
Story freshness: 51 days old; Aged out of breaking-news scoring windows; ranking reflects the durable signal from the full source set.

完整方法见我们的编辑标准。

报道来源 [1]

arXiv cs.AI TIER_1 English(EN) · Kasun Dewage, Marianna Pensky, Suranadi De Silva, T. H. Bandara · 2026-08-11 04:00

Spectral Outliers Reveal Dominant Learned Structure in Transformer Attention

arXiv:2608.07921v1 Announce Type: cross Abstract: We apply Marchenko-Pastur (MP) random matrix theory to pre-trained attention weights in order to separate each projection matrix into a random-like bulk and a set of spectral outliers. We validate this decomposition causally: zero…

本文如何被排名

报道来源 [1]

Spectral Outliers Reveal Dominant Learned Structure in Transformer Attention

相关实体

相关话题