New research advances on-policy distillation for LLM training · 6 sources tracked
ByPulseAugur Editorial·[9 sources]·
Researchers are developing advanced techniques for on-policy distillation (OPD), a method used to improve large language models. New approaches like $\gamma$OPD and STRIDE aim to enhance optimization stability and efficiency by refining how temporal credit is assigned and when to stop or restart training. Other work explores data-free distillation, where models generate their own training questions, and methods like SCOPE-OPSD that leverage privileged subspaces for improved self-distillation, showing promise in areas such as mathematical reasoning and code generation.
AI
IMPACT
These advancements in on-policy distillation could lead to more efficient and effective training of large language models, improving their performance in complex reasoning tasks.
RANK_REASON
Multiple research papers published on arXiv detailing new methods and improvements for on-policy distillation.
arXiv:2609.20784v1 Announce Type: cross Abstract: Multi-turn agents trained with reinforcement learning (RL) receive a single scalar reward per trajectory, which motivates self on-policy distillation (OPD) to supply dense token-level supervision from a self-teacher with privilege…
arXiv:2609.18321v1 Announce Type: cross Abstract: Offline on-policy distillation gains efficiency by collecting student trajectories and teacher supervision once and reusing them throughout optimization. The same reuse makes imperfect supervision persistent. Since even strong tea…
Multi-turn agents trained with reinforcement learning (RL) receive a single scalar reward per trajectory, which motivates self on-policy distillation (OPD) to supply dense token-level supervision from a self-teacher with privileged task skills, letting a skill-free student intern…
Offline on-policy distillation gains efficiency by collecting student trajectories and teacher supervision once and reusing them throughout optimization. The same reuse makes imperfect supervision persistent. Since even strong teachers can fail, we ask \emph{what remains learnabl…
arXiv:2609.16937v1 Announce Type: cross Abstract: On-policy distillation (OPD) has emerged as an effective approach for large language model post-training, yet existing objectives face a trade-off between objective fidelity and optimization stability. Token-level OPD provides sta…
On-policy distillation (OPD) has emerged as an effective approach for large language model post-training, yet existing objectives face a trade-off between objective fidelity and optimization stability. Token-level OPD provides stable but local supervision, whereas sequence-level …
arXiv:2609.14636v1 Announce Type: cross Abstract: On-policy distillation (OPD) has become a standard approach for transferring capabilities from large teachers to compact students. Its cost, however, is dominated by autoregressive student rollouts and scales poorly in multi-turn …
arXiv:2609.14193v1 Announce Type: cross Abstract: On-policy distillation (OPD) has become a standard component of frontier post-training pipelines, yet how much its training data actually contributes has gone largely unexamined. On the two teacher-student pairings most common in …
arXiv cs.LG
TIER_1English(EN)·Yunmeng Chen (Chongqing Ant Consumer Finance Co., Ltd), Kunyu Wang (Alibaba Cloud Computing Co., Ltd), Peihan Li (Chongqing Ant Consumer Finance Co., Ltd), Yi Wang (Chongqing Ant Consumer Finance Co., Ltd), Shuyin Xia (Chongqing University of Posts and T…·
arXiv:2609.12579v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) scores student-generated prefixes with a solution-conditioned self-teacher, yet transfers supervision only through next-token probabilities. We ask whether the aligned final-layer discrepancy offer…
