Outcome-based RL enables transformers to reason with right data

A new paper demonstrates that transformers trained with outcome-based reinforcement learning can develop reasoning abilities, specifically by generating intermediate steps like Chain-of-Thought. The research proves that even with sparse rewards focused on final answer correctness, policy gradients can guide transformers to learn structured, iterative algorithms for tasks like graph traversal. Crucially, the study highlights that the emergence of this reasoning capability is dependent on the training data distribution, requiring a sufficient number of simpler examples to generalize effectively. AI

IMPACT Demonstrates a theoretical pathway for emergent reasoning in LLMs, potentially guiding future training methodologies for improved performance on complex tasks.

RANK_REASON Academic paper detailing a theoretical and experimental analysis of transformer reasoning capabilities. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Chain-of-Thought
• Reinforcement Learning
• Transformer
• Yuval Ran-Milo

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →