研究人员推出了一种名为 Filtered Truncated Importance Sampling (F-TIS) 的新训练范式，专为像 GRPO 这样的从人类反馈中强化学习 (RLHF) 方法设计。F-TIS 解决了使用异构模型进行训练的挑战，在这种情况下，不同的模型在同一任务上协作，这通常会导致离策略样本，从而阻碍收敛。所提出的框架允许不同的模型高效地协同工作，保持通信并实现与在线策略训练相当的收敛性。在某些场景下，F-TIS 甚至在分布外任务上表现出更好的泛化能力，性能提升高达 12%。 AI
影响 支持更灵活、更高效的异构 LLM 协作训练，可能提高泛化能力。
排序理由 发布了一篇详细介绍 LLM 新训练方法的学术论文。
AI 生成摘要 · Google Gemini · 来自 2 个来源。 我们如何撰写摘要 →
arXiv:2605.22537v1 Announce Type: new Abstract: Reinforcement learning methods such as GRPO have seen great popularity in LLM post-training. In GRPO, models produce completions to a set of prompts, which are rewarded, and the policy is updated towards the relatively high reward c…
Reinforcement learning methods such as GRPO have seen great popularity in LLM post-training. In GRPO, models produce completions to a set of prompts, which are rewarded, and the policy is updated towards the relatively high reward completions. Due to the auto-regressive nature of…