研究人员开发了一种贝叶斯理论来解释Transformer注意力机制中“复制头”的涌现。他们对单层softmax注意力网络的分析揭示了这些注意力模式形成的相变，这种相变取决于训练数据的量。该理论框架为特定子电路的突然出现提供了第一性原理的解释，类似于在大语言模型训练中的观察结果。 AI
影响 为LLM中涌现行为提供了理论解释，可能指导未来的模型设计和训练。
排序理由 该集群包含一篇学术论文，详细介绍了理解Transformer模型中特定机制的新理论框架。
AI 生成摘要 · Google Gemini · 来自 2 个来源。 我们如何撰写摘要 →
arXiv:2606.12058v1 Announce Type: new Abstract: Attention is the key mechanism underlying in-context learning in transformers, and attention patterns have been observed empirically to emerge abruptly during training. We present a Bayesian theory of feature learning in attention; …
Attention is the key mechanism underlying in-context learning in transformers, and attention patterns have been observed empirically to emerge abruptly during training. We present a Bayesian theory of feature learning in attention; we then focus on how the copy subcircuit in the …