PulseAugur
中
实时 04:13:27

New methods enhance on-policy distillation for LLMs

作者 PulseAugur 编辑部 · [5 个来源] ·

Researchers have developed new methods to improve the efficiency and stability of on-policy distillation (OPD) for large language models. One approach, vOPD, uses a control variate baseline derived from the reverse KL divergence to reduce gradient variance without significant computational overhead. Another method, ROPD, enables rubric-based distillation using only teacher-generated responses, offering a black-box compatible alternative to logit-based OPD. A third technique, Near-Policy Distillation (NPD), accelerates the process through asynchronous generation and selective packing, achieving substantial speedups and outperforming standard fine-tuning. AI

影响 These advancements offer more efficient and stable methods for aligning LLMs, potentially accelerating their deployment in complex reasoning tasks.

排序理由 Multiple arXiv papers introduce novel methods for improving on-policy distillation techniques in LLMs.

在 arXiv cs.CL 阅读 →

AI 生成摘要 · Google Gemini · 来自 5 个来源。 我们如何撰写摘要 →

New methods enhance on-policy distillation for LLMs

报道来源 [5]

  1. arXiv cs.CL TIER_1 English(EN) · Tomas Pfister ·

    RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards

    Training deep research agents, namely systems that plan, search, evaluate evidence, and synthesize long-form reports, pushes reinforcement learning beyond the regime of verifiable rewards. Their outputs lack ground-truth answers, their trajectories span many tool-augmented decisi…

  2. arXiv cs.AI TIER_1 English(EN) · Yohan Jo ·

    KL for a KL: On-Policy Distillation with Control Variate Baseline

    On-Policy Distillation (OPD) has emerged as a dominant post-training paradigm for large language models, especially for reasoning domains. However, OPD remains unstable in practice due to the high gradient variance of its single-sample Monte Carlo estimator, and recipes for stabl…

  3. arXiv cs.LG TIER_1 English(EN) · Tat-Seng Chua ·

    Rubric-based On-policy Distillation

    On-policy distillation (OPD) is a powerful paradigm for model alignment, yet its reliance on teacher logits restricts its application to white-box scenarios. We contend that structured semantic rubrics can serve as a scalable alternative to teacher logits, enabling OPD using only…

  4. arXiv cs.LG TIER_1 English(EN) · Miao Rang, Zhenni Bi, Hang Zhou, Kai Han, Xuechun Wang, An Xiao, Xinghao Chen, Yunhe Wang, Hanting Chen ·

    Near-Policy: Accelerating On-Policy Distillation via Asynchronous Generation and Selective Packing

    arXiv:2605.05940v1 Announce Type: new Abstract: Standard knowledge distillation for autoregressive models often suffers from distribution mismatch. While on-policy methods mitigate this by leveraging student-generated outputs, they rely on computationally expensive Reinforcement …

  5. arXiv cs.CL TIER_1 English(EN) · Hanting Chen ·

    Near-Policy: Accelerating On-Policy Distillation via Asynchronous Generation and Selective Packing

    Standard knowledge distillation for autoregressive models often suffers from distribution mismatch. While on-policy methods mitigate this by leveraging student-generated outputs, they rely on computationally expensive Reinforcement Learning (RL) frameworks. To improve efficiency,…

相关实体

相关话题