New RL method boosts LLM arithmetic reasoning with dense rewards

Researchers have developed a new reinforcement learning (RL) method called Bellman-Consistent Trajectory RL to improve LLM arithmetic reasoning, particularly when dealing with sparse rewards. This framework uses Trajectory Bellman Residual Minimization (TBRM) to convert sparse rewards into dense, token-level signals. By integrating Reinforce Leave-One-Out (RLOO) for reduced gradient variance and hindsight goal relabeling for faster learning, the TBRM-RLOO approach significantly enhances performance on arithmetic tasks. The method achieves state-of-the-art results on the Countdown task, boosting Pass@1 accuracy to 0.55-0.64 and Pass@16 accuracy to 76.0-84.0%, outperforming existing baselines and demonstrating high single-sample efficiency. AI

IMPACT Enhances LLM capabilities in complex reasoning tasks, potentially improving performance in areas requiring mathematical problem-solving.

RANK_REASON Academic paper detailing a new method for LLM arithmetic reasoning. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Bellman-Consistent Trajectory RL
• Countdown task
• Direct Preference Optimization
• GRPO
• Hugging Face
• LLM Arithmetic Reasoning
• Reinforce Leave-One-Out
• supervised fine-tuning
• TBRM-RLOO
• Trajectory Bellman Residual Minimization

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →