English(EN) Exact Fusion of Softmax Attention Heads Can Require Exponentially Many Atoms

神经网络中的注意力头融合可能需要指数级原子

作者 PulseAugur 编辑部 · [1 个来源] · 2026-11-02 11:10

一篇新发表在arXiv上的论文探讨了神经网络中注意力头融合的计算复杂性。研究表明，组合独立归一化的注意力头会显著增加维持其输出所需的显式键值条目（即原子）的数量。具体而言，该论文证明了函数W_d(q)存在一个严格分离，表明其d个双原子头需要2d个头原子项，而单头表示则需要2^d个原子。 AI

影响这项研究突显了注意力机制中潜在的计算瓶颈，表明某些融合策略可能导致资源需求呈指数级增长。

排序理由该集群包含一篇发表在arXiv上的研究论文，详细介绍了关于神经网络架构的理论发现。[lever_c_demoted from research: ic=1 ai=1.0]

AI 生成摘要 · Google Gemini · 来自 1 个来源。我们如何撰写摘要 →

本文如何被排名

Signal score: 2 / 100; Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket: Tool; 该集群包含一篇发表在arXiv上的研究论文，详细介绍了关于神经网络架构的理论发现。[lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration: Single-source cluster; Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics: paper, infra; Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance: High; Clearly on-topic for AI-industry coverage.
Story freshness: 1 days old; Coverage has settled into its steady-state source set.

完整方法见我们的编辑标准。

arXiv cs.NE (Neural & Evolutionary) TIER_1 English(EN) · Naohiko Matsuda · 2026-11-02 11:10

Exact Fusion of Softmax Attention Heads Can Require Exponentially Many Atoms

Fusing independently normalized attention heads can increase the number of explicit key-value entries, or atoms, needed to preserve their output. We prove a sharp separation for $W_d(q)=\sum_{j=1}^dσ(q_j)$: its $d$ two-atom heads use $2d$ head-atom terms, whereas its minimum sing…