Researchers have developed a new pipeline called OnQ to improve the reasoning capabilities of low-bit large language models (LLMs) that have undergone quantization-aware training (QAT). Existing QAT methods often struggle with long-horizon reasoning and token budget limitations. OnQ addresses these issues by using on-policy distillation, where the quantized model generates its own prefixes and receives supervision from a full-precision teacher model. Additionally, it employs Dynamic KL to balance the correction of under- and overpredictions by weighting KL divergence based on candidate probability ratios. Experiments on Qwen3 models at 2- and 3-bit precision demonstrated that OnQ significantly outperforms existing baselines across nine benchmarks, particularly in long-horizon reasoning tasks, and produces shorter, less repetitive responses. AI
IMPACT This research could lead to more efficient and capable LLMs by improving performance on quantized models, potentially reducing computational costs and enabling wider deployment.
RANK_REASON The cluster contains an academic paper detailing a new method for improving LLM performance. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
