Researchers have introduced MOER, a novel framework for mix-policy optimization in reinforcement learning that integrates expert guidance with self-exploration. MOER employs a decaying expert chain-of-thought prefix to guide model reasoning while progressively annealing the prefix length to foster self-exploration. A selective rollback mechanism ensures the correctness of off-policy trajectories by replacing failed completions with expert traces. The framework utilizes a harmonized mix-policy objective, combining GRPO loss for on-policy tokens and a trust-region SFT loss for both expert prefixes and completions, which significantly stabilizes training. Experiments demonstrate MOER's superior performance in mathematical and general domain reasoning tasks, outperforming existing baselines across various tasks, model scales, and base models. AI
IMPACT This research could lead to more robust and capable AI models in reasoning tasks by improving the balance between guided learning and self-exploration.
RANK_REASON The cluster describes a new research paper detailing a novel framework for AI model training. [lever_c_demoted from research: ic=1 ai=1.0]
- alphaXiv
- arXiv
- CatalyzeX
- DagsHub
- Gotit.pub
- Grpo
- Hugging Face
- IArxiv
- MOER
- ScienceCast
- supervised fine-tuning
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
