研究人员在语言模型中发现了一种称为“自然遗忘”的现象，即学到的规则可以在训练过程中消失，而损失曲线没有任何变化。这种遗忘与规则在训练数据中出现的频率直接相关；出现频率较低的规则更容易被竞争模式覆盖。有趣的是，这个过程是不对称的：虽然外部干预可以轻易破坏学到的规则，但重新引入支持数据并不能可靠地恢复它。 AI
影响 这项研究突显了当前LLM训练中的一个关键漏洞，表明模型可能无法可靠地保留学到的知识，从而影响其长期效用和安全性。
排序理由 该集群包含一篇研究论文，详细介绍了在AI模型中观察到的一种新现象。[lever_c_demoted from research: ic=1 ai=1.0]
AI 生成摘要 · Google Gemini · 来自 3 个来源。 我们如何撰写摘要 →
arXiv:2606.26050v1 Announce Type: cross Abstract: Midway through an ordinary pretraining run, a small language model learns the pronoun-gender rule: cued with a girl's name ("Sue cried because"), it resolves the next pronoun to she, generalizing to held-out probes (0.94 by step 9…
Midway through an ordinary pretraining run, a small language model learns the pronoun-gender rule: cued with a girl's name ("Sue cried because"), it resolves the next pronoun to she, generalizing to held-out probes (0.94 by step 925). By step 3,500 the same model scores near zero…
Midway through an ordinary pretraining run, a small language model learns the pronoun-gender rule: cued with a girl's name ("Sue cried because"), it resolves the next pronoun to she, generalizing to held-out probes (0.94 by step 925). By step 3,500 the same model scores near zero…