一项新的研究论文介绍了“自然遗忘”（natural ungrokking）的概念，描述了语言模型如何在预训练期间学习一条规则，却在没有损失曲线变化的情况下随后将其遗忘。研究发现，已学规则的存续取决于它们在训练数据中出现的频率，而非数据与参数的比例。有趣的是，该研究还表明，虽然可以有意地破坏一条已学规则，但恢复它却是一个不对称的过程，即使在支持性数据显著增加的情况下也未观察到恢复。 AI
影响 这项研究突显了大型语言模型训练中一个潜在的脆弱性，表明已学行为可能会在没有明确指示的情况下丢失，从而影响模型的可靠性和可解释性。
排序理由 该集群包含一篇详细介绍语言模型训练中一种现象的研究论文。
AI 生成摘要 · Google Gemini · 来自 3 个来源。 我们如何撰写摘要 →
arXiv:2606.26050v1 Announce Type: cross Abstract: Midway through an ordinary pretraining run, a small language model learns the pronoun-gender rule: cued with a girl's name ("Sue cried because"), it resolves the next pronoun to she, generalizing to held-out probes (0.94 by step 9…
Midway through an ordinary pretraining run, a small language model learns the pronoun-gender rule: cued with a girl's name ("Sue cried because"), it resolves the next pronoun to she, generalizing to held-out probes (0.94 by step 925). By step 3,500 the same model scores near zero…
Midway through an ordinary pretraining run, a small language model learns the pronoun-gender rule: cued with a girl's name ("Sue cried because"), it resolves the next pronoun to she, generalizing to held-out probes (0.94 by step 925). By step 3,500 the same model scores near zero…