PulseAugur
EN
LIVE 03:54:52

New self-distillation methods boost LLM performance on reasoning tasks

By PulseAugur Editorial · [5 sources] ·

Researchers have developed new self-distillation techniques for large language models to improve their performance without relying on external feedback. AVSD (Adaptive-View Self-Distillation) balances consensus signals across multiple privileged information views with view-specific residuals to enhance learning. Self-Policy Distillation (SPD) extracts a capability subspace from gradients to improve performance and generalizability, particularly in code generation and mathematical reasoning. CEPO (Contrastive Evidence Policy Optimization) sharpens credit assignment at decisive tokens by contrasting correct answers with incorrect ones, improving accuracy on multimodal mathematical reasoning benchmarks. AI

IMPACT These self-distillation techniques offer improved performance and generalizability for LLMs in complex reasoning tasks without external supervision.

RANK_REASON The cluster contains multiple research papers detailing novel methods for self-distillation in large language models.

Read on arXiv cs.CL →

AI-generated summary · Google Gemini · from 5 sources. How we write summaries →

New self-distillation methods boost LLM performance on reasoning tasks

COVERAGE [5]

  1. arXiv cs.AI TIER_1 English(EN) · Duy Nguyen, Hanqi Xiao, Archiki Prasad, Zaid Khan, Anirban Das, Austin Zhang, Sambit Sahu, Hyunji Lee, Elias Stengel-Eskin, Mohit Bansal ·

    AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals

    arXiv:2605.20643v1 Announce Type: cross Abstract: Self-distillation enables language models to learn on-policy from their own trajectories by using the same model as both student and teacher, with the teacher being conditioned on privileged information unavailable to the student.…

  2. arXiv cs.CL TIER_1 English(EN) · Guangya Hao, Yitong Shang, Yunbo Long, Zhuokai Zhao, Hanxue Liang ·

    Self-Policy Distillation via Capability-Selective Subspace Projection

    arXiv:2605.22675v1 Announce Type: new Abstract: Self-distillation bootstraps large language models (LLMs) by training on their own generations. However, existing methods either rely on external signals to curate self-generated outputs (e.g., correctness filtering, execution feedb…

  3. arXiv cs.CL TIER_1 English(EN) · Hanxue Liang ·

    Self-Policy Distillation via Capability-Selective Subspace Projection

    Self-distillation bootstraps large language models (LLMs) by training on their own generations. However, existing methods either rely on external signals to curate self-generated outputs (e.g., correctness filtering, execution feedback, and reward search), which are costly and un…

  4. arXiv cs.AI TIER_1 English(EN) · Mohit Bansal ·

    AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals

    Self-distillation enables language models to learn on-policy from their own trajectories by using the same model as both student and teacher, with the teacher being conditioned on privileged information unavailable to the student. Such information can come in different types or v…

  5. arXiv cs.CL TIER_1 English(EN) · Salman Khan ·

    CEPO: RLVR Self-Distillation using Contrastive Evidence Policy Optimization

    When a model produces a correct solution under reinforcement learning with verifiable rewards (RLVR), every token receives the same reward signal regardless of whether it was a decisive reasoning step or a grammatical filler. A natural fix is to condition the model on the correct…

RELATED ENTITIES

RELATED TOPICS