研究人员开发了一个分析模型，用于解释训练任务多样性如何影响Transformer中的上下文学习（ICL）。该模型将训练任务向量视为低秩高斯分布，并证明了以非重叠子空间列定义的任务多样性可以增强ICL的泛化和优化能力。该框架有助于解释为何多样化训练可以缩短ICL平台期并实现分布外泛化，其研究结果也适用于非线性Transformer。 AI
影响 提供了一个理论框架，用于理解和潜在地改进Transformer的ICL能力。
排序理由 该集群包含一篇预印本学术论文，详细介绍了一个用于Transformer行为的新分析模型。
AI 生成摘要 · Google Gemini · 来自 2 个来源。 我们如何撰写摘要 →
arXiv:2606.06814v1 Announce Type: new Abstract: The transformer's emergent ability to perform in-context learning (ICL) has sparked a wide range of studies designed to understand its underlying mechanisms. Existing works often study how training task diversity, defined either as …
The transformer's emergent ability to perform in-context learning (ICL) has sparked a wide range of studies designed to understand its underlying mechanisms. Existing works often study how training task diversity, defined either as the number of ICL training task vectors or as th…