English(EN) Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening

新的PPO方法解决了LLM强化学习中的价值平坦化问题

作者 PulseAugur 编辑部 · [1 个来源] · 2026-09-17 04:00

研究人员发现Proximal Policy Optimization (PPO) 中存在一种称为价值平坦化（Value Flattening）的故障模式，在这种模式下，即使实际值急剧变化，Critic估计的状态值在中间状态下也会变得平坦。这种现象在更大的状态空间中会加剧，并可能导致冗余更新。为了解决这个问题，该论文引入了稀疏近端策略优化（SParse Proximal Policy Optimization, SP$^3$O），它仅对每次响应中少数分隔良好的状态应用价值损失。在Qwen3-Base上的实验表明，SP$^3$O能有效缓解价值平坦化并改善策略学习。 AI

影响通过解决Critic学习中的特定故障模式，引入了一种改进大型语言模型策略学习的方法。

排序理由详细介绍LLM强化学习新方法的学术论文。[lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.AI 阅读 →

AI 生成摘要 · Google Gemini · 来自 1 个来源。我们如何撰写摘要 →

本文如何被排名

Signal score: 16 / 100; Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket: Tool; 详细介绍LLM强化学习新方法的学术论文。[lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration: Single-source cluster; Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics: paper, model release; Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance: High; Clearly on-topic for AI-industry coverage.
Story freshness: Breaking (< 6h); Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

完整方法见我们的编辑标准。

报道来源 [1]

arXiv cs.AI TIER_1 English(EN) · Yizhuo Li, Jianhao Yan, Yun Luo, Zhi Wang, Futing Wang, Rong-Xi Tan, Kanghui Tian, Ganqu Cui, Ning Ding, Peilin Zhao, Yafu Li, Yu Cheng · 2026-09-17 04:00

重新思考PPO中的Critic学习：理解和缓解价值平坦化

arXiv:2609.18708v1 Announce Type: cross Abstract: In reinforcement learning for large language models, Proximal Policy Optimization (PPO) commonly uses a critic to estimate state values and reduce the variance of policy updates. However, we uncover a systematic failure mode in PP…

本文如何被排名

报道来源 [1]

重新思考PPO中的Critic学习：理解和缓解价值平坦化

相关实体

相关话题