研究人员推出了一些新颖的语言模型自我蒸馏技术，旨在提高性能，而无需外部教师或真实标签。激活条件自我蒸馏（ACSD）从模型激活中提取引导向量来指导学习，在数学和编码基准测试中取得了高精度。另一种方法，知识到提示（K2P），从教师解决方案中合成和优化可重用的指令，用于无标签蒸馏。此外，InFlow将该过程建模为信息流，根据信念转移检索和选择信息源，以增强策略内自我蒸馏。 AI
影响 这些方法通过减少对外部监督和教师模型的依赖，为提高语言模型的能力和效率提供了途径。
排序理由 多篇研究论文介绍了语言模型的新型自我蒸馏技术。
AI 生成摘要 · Google Gemini · 来自 4 个来源。 我们如何撰写摘要 →
完整方法见我们的编辑标准。
arXiv:2609.38342v1 Announce Type: new Abstract: On-policy self-distillation uses a model as its own teacher to provide dense supervision for reasoning, often through reference-solution conditioning. Providing privileged information does not by itself ensure effective token-level …
arXiv:2609.39494v1 Announce Type: new Abstract: Self-distillation with privileged context adapts a language model from demonstrations by letting the model, once conditioned on a reference response, teach its context-free copy token by token. Our taxonomy reveals existing methods …
arXiv:2609.38898v1 Announce Type: cross Abstract: Knowledge distillation can transfer reasoning from stronger teachers to frozen students through reusable prompts, but avoiding weight updates does not eliminate supervision. Without ground-truth answers, teacher solutions are unve…
arXiv:2609.36695v1 Announce Type: new Abstract: Self-distillation turns knowledge distillation into a closed learning loop and offers a path toward recursive self-improvement. Without an external teacher, however, the model must determine both what information can improve its sup…