New research analyzes how SFT, RL, and OPD shape LLM reasoning confidence

A new research paper introduces a three-stage framework to analyze how supervised fine-tuning (SFT), reinforcement learning (RL), and on-policy distillation (OPD) affect the confidence calibration of large language models during reasoning. The study, conducted on mathematical reasoning benchmarks, found that OPD is most effective for pre-reasoning confidence, SFT excels at early stopping, and RL provides the most reliable signal for answer aggregation. The researchers also developed PosConf, a position-aware confidence strategy that improves RL answer aggregation and OPD early stopping by leveraging confidence signals from reliable relative-position intervals. AI

IMPACT This research could lead to more reliable and efficient LLM reasoning by improving how confidence is measured and utilized during generation.

RANK_REASON The cluster contains an academic paper detailing a new analysis framework and method for large language models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• on-policy distillation
• PosConf
• reinforcement learning
• Shuhao Li
• supervised fine-tuning

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →