PulseAugur
EN
LIVE 06:34:25

New methods enhance on-policy distillation for AI model training · 6 sources tracked

By PulseAugur Editorial · [6 sources] ·

Researchers are developing new methods for on-policy distillation, a technique used to train smaller AI models by having them learn from the outputs of larger, more capable models. Apple Machine Learning Research has introduced a diagnostic framework to analyze where and why on-policy distillation is effective, finding that the signal is more beneficial on incorrect student rollouts. Meanwhile, new methods like Veto and RG-OPD aim to stabilize training by reformulating the distillation objective or using verifier feedback to filter unreliable teacher signals. Additionally, ReOPD offers an off-environment approach that reuses pre-collected trajectories to speed up multi-turn distillation for agentic tasks. AI

IMPACT These advancements in on-policy distillation could lead to more efficient training of smaller, capable AI models, potentially reducing computational costs and improving performance on complex reasoning and agentic tasks.

RANK_REASON Multiple research papers introducing new methods and analyses for on-policy distillation.

Read on Hugging Face Daily Papers →

AI-generated summary · Google Gemini · from 6 sources. How we write summaries →

New methods enhance on-policy distillation for AI model training · 6 sources tracked

COVERAGE [6]

  1. Apple Machine Learning Research TIER_1 English(EN) ·

    Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why

    On-policy distillation offers dense, per-token supervision for training reasoning models; however, it remains unclear under which conditions this signal is beneficial and under which it is detrimental. Which teacher model should be used, and in the case of self-distillation, whic…

  2. arXiv cs.AI TIER_1 English(EN) · Ijun Jang, Jewon Yeom, Juan Yeo, Hyunggyu Lim, Taesup Kim ·

    Stable On-Policy Distillation through Adaptive Target Reformulation

    arXiv:2601.07155v3 Announce Type: replace-cross Abstract: Knowledge distillation (KD) is a widely adopted technique for transferring knowledge from large language models to smaller student models; however, conventional supervised KD often suffers from a distribution mismatch betw…

  3. arXiv cs.AI TIER_1 English(EN) · Mohammad Sadegh Akhondzadeh, Vijay Lingam, Atula Tejaswi, Chanakya Ekbote, Sujay Sanghavi, Aleksandar Bojchevski ·

    Reward-Gated On-Policy Distillation

    arXiv:2607.04037v1 Announce Type: cross Abstract: On-policy distillation is a powerful way to transfer reasoning ability from a strong teacher to a smaller student: the student samples trajectories from its own policy, and the teacher provides dense token-level supervision on the…

  4. Hugging Face Daily Papers TIER_1 English(EN) ·

    Multi-Turn On-Policy Distillation with Prefix Replay

    We study on-policy distillation (OPD) for agentic tasks, where an LLM agent interacts with an environment over multiple turns and a student imitates a teacher over these multi-turn interaction histories. Fully online OPD is costly because each update requires fresh student rollou…

  5. arXiv stat.ML TIER_1 English(EN) · Baohao Liao, Hanze Dong, Christof Monz, Xinxing Xu, Li Dong, Furu Wei ·

    Multi-Turn On-Policy Distillation with Prefix Replay

    arXiv:2607.04763v1 Announce Type: cross Abstract: We study on-policy distillation (OPD) for agentic tasks, where an LLM agent interacts with an environment over multiple turns and a student imitates a teacher over these multi-turn interaction histories. Fully online OPD is costly…

  6. arXiv stat.ML TIER_1 English(EN) · Furu Wei ·

    Multi-Turn On-Policy Distillation with Prefix Replay

    We study on-policy distillation (OPD) for agentic tasks, where an LLM agent interacts with an environment over multiple turns and a student imitates a teacher over these multi-turn interaction histories. Fully online OPD is costly because each update requires fresh student rollou…

RELATED ENTITIES

RELATED TOPICS