New SPRO framework enhances LLM reasoning with intrinsic rewards

Researchers have introduced Self-Guided Process Reward Optimization (SPRO), a new framework designed to improve the reasoning abilities of large language models (LLMs) through process-aware reinforcement learning. SPRO addresses the computational overhead of traditional process reward models by deriving rewards intrinsically from the policy model itself and redefining step-wise advantage with Cumulative Process Rewards (CPR) and Masked Step Advantage (MSA). Experiments show SPRO achieves 3.4x higher training efficiency and a 12.9% improvement in test accuracy compared to vanilla GRPO, while also maintaining stable policy entropy and reducing response length. AI

IMPACT This framework could lead to more efficient training of LLMs for complex reasoning tasks, potentially reducing computational costs and improving performance.

RANK_REASON The cluster contains a research paper detailing a new framework for process reinforcement learning. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• Cumulative Process Rewards
• GRPO
• Hao Kong
• large-language models
• Masked Step Advantage
• Process Reinforcement Learning

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →