New method trains LLMs to refuse futile reasoning

Researchers have identified a phenomenon called "futile reasoning" in large language models, where models generate lengthy, complex, but ultimately incorrect derivations on tasks beyond their capabilities. This often leads to plausible-sounding but erroneous outputs that can mislead users. To combat this, a new method called CaRL (Capability-aligned Reinforcement Learning) has been developed. CaRL uses reward shaping to encourage models to refuse impossible tasks and hindsight refusal augmentation to train models to recognize and refuse futile reasoning, thereby aligning model behavior with its actual capabilities without sacrificing utility. AI

IMPACT This research could lead to more reliable LLMs that avoid generating misleading information on complex tasks.

RANK_REASON Academic paper detailing a new method for LLM training. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• alphaXiv
• arXiv
• CaRL
• CatalyzeX
• DagsHub
• Gotit.pub
• Hugging Face
• Large language models
• ScienceCast

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →