研究人员开发了一个统一的统计学习框架，用于分析因果和掩码语言模型的上下文学习（ICL）能力。该框架将自回归和掩码预训练目标置于共同的超额风险分析中，为两者提供了理论界限。实验表明，像掩码对编码器（MPE）这样的掩码语言模型可以达到与GPT-2风格的因果Transformer相当的性能，这表明ICL并非因果模型的专属。 AI
影响 这项研究可能有助于更好地理解和开发不同类型语言模型的上下文学习能力。
排序理由 该集群包含一篇详细介绍分析语言模型能力新理论框架的学术论文。
AI 生成摘要 · Google Gemini · 来自 2 个来源。 我们如何撰写摘要 →
arXiv:2607.04081v1 Announce Type: cross Abstract: In-context learning (ICL) has emerged as a central capability of pretrained language models, yet its theoretical analysis has focused primarily on causal language models trained by left-to-right autoregressive prediction, such as …
In-context learning (ICL) has emerged as a central capability of pretrained language models, yet its theoretical analysis has focused primarily on causal language models trained by left-to-right autoregressive prediction, such as GPT-style models. Masked language models instead r…