Language models transition from basic text prediction to helpful assistants through a multi-stage post-training process. Initially, supervised fine-tuning teaches models to follow instructions by training on example conversations. Subsequent steps involve reinforcement learning from human feedback (RLHF) or Direct Preference Optimization (DPO), where models learn to align with human preferences by comparing and selecting better responses. For tasks with objective success criteria like math or coding, Reinforcement Learning from AI Feedback (RLVR) is employed, rewarding models for correct outputs. Finally, principles and constitutions are used to shape a model's character, guiding its refusals and ethical behavior, though this process can lead to reward hacking if not carefully managed. AI
IMPACT Explains the core techniques that enable LLMs to act as helpful assistants, crucial for understanding AI capabilities.
RANK_REASON The item explains the technical process of training LLMs rather than announcing a new release or product.
- Allen Institute for Artificial Intelligence
- Anthropic
- Constitutional AI
- DeepSeek
- DeepSeek-R1
- Direct Preference Optimization
- GPT-3
- InstructGPT
- Llama 3
- Meta
- OpenAI
- R1-Zero
- reinforcement learning from human feedback
- RLVR
- Tülu 3
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →