研究人员开发了新方法来提高大型语言模型On-Policy蒸馏（OPD）的效率和稳定性。一种方法vOPD使用源自反向KL散度的控制变量基线，在没有显著计算开销的情况下降低梯度方差。另一种方法ROPD仅使用教师生成的响应即可实现基于规则的蒸馏，提供了基于logit的OPD的黑盒兼容替代方案。第三种技术Near-Policy Distillation（NPD）通过异步生成和选择性打包来加速该过程，实现了显著的加速并优于标准微调。 AI
影响 这些进展为对齐LLM提供了更有效、更稳定的方法，有可能加速它们在复杂推理任务中的部署。
排序理由 多篇arXiv论文介绍了改进LLM中On-Policy蒸馏技术的新方法。
AI 生成摘要 · Google Gemini · 来自 5 个来源。 我们如何撰写摘要 →
Training deep research agents, namely systems that plan, search, evaluate evidence, and synthesize long-form reports, pushes reinforcement learning beyond the regime of verifiable rewards. Their outputs lack ground-truth answers, their trajectories span many tool-augmented decisi…
On-Policy Distillation (OPD) has emerged as a dominant post-training paradigm for large language models, especially for reasoning domains. However, OPD remains unstable in practice due to the high gradient variance of its single-sample Monte Carlo estimator, and recipes for stabl…
On-policy distillation (OPD) is a powerful paradigm for model alignment, yet its reliance on teacher logits restricts its application to white-box scenarios. We contend that structured semantic rubrics can serve as a scalable alternative to teacher logits, enabling OPD using only…
arXiv:2605.05940v1 Announce Type: new Abstract: Standard knowledge distillation for autoregressive models often suffers from distribution mismatch. While on-policy methods mitigate this by leveraging student-generated outputs, they rely on computationally expensive Reinforcement …
Standard knowledge distillation for autoregressive models often suffers from distribution mismatch. While on-policy methods mitigate this by leveraging student-generated outputs, they rely on computationally expensive Reinforcement Learning (RL) frameworks. To improve efficiency,…