LLM capabilities primarily stem from imitative learning, not RL, analysis suggests

A recent analysis argues that the capabilities of large language models (LLMs) are primarily derived from imitative learning, such as pre-training and supervised fine-tuning, rather than reinforcement learning (RL). While RL, including RL from human feedback (RLHF) and RL from AI feedback (RLAIF), plays a role, its contribution to LLM capabilities is significantly smaller than imitative learning. This perspective suggests that the efficiency of RL in imparting capabilities is orders of magnitude lower than imitative learning, impacting how we understand model legibility and alignment. AI

IMPACT This perspective challenges conventional understanding of LLM training, potentially influencing future research directions and resource allocation in AI development.

RANK_REASON The item is an analysis and opinion piece on LLM training methodologies, not a primary release or research finding.

Read on LessWrong (AI tag) →

• Dwarkesh Patel
• imitative learning
• pre-training
• reinforcement learning
• reinforcement learning from AI feedback
• reinforcement learning from human feedback
• Reinforcement Learning from Verifiable Rewards
• RL is even more information inefficient than you thought
• RLVR
• supervised fine-tuning
• The Extreme Inefficiency of RL for Frontier Models
• Toby Ord

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →