New self-distillation method boosts reasoning models without privileged data

Researchers have developed a new self-distillation technique called Weakening-Induced On-Policy Self-Distillation (WISD) to improve large reasoning models. Unlike previous methods that require privileged data, WISD creates a weaker version of the model through quantization or pruning. The contrast between the original and weaker models generates training signals, enhancing performance on mathematical and coding tasks. This approach has shown significant accuracy improvements, for example, boosting Qwen3.5-4B's accuracy by over 12 percentage points on the HMMT 2025 benchmark. AI

IMPACT This technique offers a novel way to enhance reasoning models without requiring external datasets, potentially making advanced model improvement more accessible.

RANK_REASON Academic paper detailing a new method for improving LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on Hugging Face Daily Papers →

• HMMT 2025
• Hugging Face
• LiveCodeBench v6
• Qwen3.5-4B
• Qwen3.6-35B-A3B
• Weakening-Induced On-Policy Self-Distillation

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →