一篇新论文分析了 Transformer 模型中的表征坍塌，挑战了之前关于 MLP 和层归一化作用的发现。研究阐明，虽然层归一化保留了仿射秩，但残差连接可以在没有 MLP 的情况下防止秩坍塌。该论文还确定了多头注意力中一个独特的头-通道不可识别问题，并提出了一种位置门控输出投影作为部分解决方案。 AI
影响 提供了对 Transformer 架构限制和潜在补救措施更精确的理解。
排序理由 学术论文，分析 Transformer 架构和表征坍塌。
AI 生成摘要 · Google Gemini · 来自 2 个来源。 我们如何撰写摘要 →
arXiv:2604.23681v1 Announce Type: cross Abstract: A widely cited result by Dong et al. (2021) showed that Transformers built from self-attention alone, without skip connections or feed-forward layers, suffer from rapid rank collapse: all token representations converge to a single…
A widely cited result by Dong et al. (2021) showed that Transformers built from self-attention alone, without skip connections or feed-forward layers, suffer from rapid rank collapse: all token representations converge to a single direction. The proposed remedy was the MLP. We sh…