New training method teaches LLMs to abstain when uncertain

Researchers have developed a new training method called Reinforced Hesitation (RH) to make language models more trustworthy by teaching them to abstain from answering when uncertain. Unlike traditional methods that reward any answer, RH uses ternary rewards, penalizing incorrect answers more severely than abstentions. Experiments on logic puzzles, medical questions, and advanced math problems demonstrated that RH effectively trains models to calibrate their honesty, with different penalty levels producing models optimized for various risk tolerances. The research also introduced inference strategies like cascading and self-cascading to leverage abstention as a coordination signal, outperforming majority voting with lower computational costs. AI

IMPACT This research could lead to more reliable and trustworthy AI systems by enabling them to accurately signal uncertainty, reducing the impact of hallucinations in critical applications.

RANK_REASON Research paper introducing a novel training methodology for language models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• alphaXiv
• CatalyzeX Code Finder for Papers
• DagsHub
• Gotit.pub
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark
• GSM8K
• Honesty over Accuracy: Trustworthy Language Models through Reinforced Hesitation
• Hugging Face
• MATH Levels 4--5
• MedQA
• Mohamad Amin Mohamadi
• Reinforced Hesitation
• Reinforcement Learning from Verifiable Rewards
• ScienceCast

• paper
• safety
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →