研究人员开发了G2D，一个三阶段流水线，结合了GRPO和DPO，以更有效地对语言模型进行离线偏好优化。该方法包括简短的GRPO预热，然后构建静态偏好数据集，最后用DPO进行微调。在Qwen2.5-7B和Llama-3.1-8B模型上的实验表明，G2D通过关注偏好数据的有效性而非仅仅数量，能够以显著降低的计算成本匹配或超越完全在线GRPO的性能。 AI
影响 通过提高数据有效性，为语言模型训练提供了比在线强化学习更节省计算资源的选择。
排序理由 该集群包含一篇详细介绍语言模型优化新方法的学术论文。
AI 生成摘要 · Google Gemini · 来自 2 个来源。 我们如何撰写摘要 →
arXiv:2605.21266v1 Announce Type: cross Abstract: Reinforcement Learning from Verifiable Rewards (RLVR) has emerged as a powerful paradigm for reasoning in language models, with GRPO as its primary example. However, GRPO requires continuous online rollout generation, making it co…
Reinforcement Learning from Verifiable Rewards (RLVR) has emerged as a powerful paradigm for reasoning in language models, with GRPO as its primary example. However, GRPO requires continuous online rollout generation, making it computationally expensive and difficult to scale. Wh…