English(EN) Every Greedy Metric Said the Model Was Improving. Then pass@64 Fell From 0.83 to 0.19

LLM 评估指标未能检测到关键性能崩溃

作者 PulseAugur 编辑部 · [1 个来源] · 2026-09-05 05:08

一项关于大型语言模型中人类反馈强化学习（RLHF）的研究揭示了评估指标的一个关键缺陷。虽然像贪婪准确率这样的标准指标显示出改进，但一个更强大的指标 pass@64 显示性能急剧下降。这表明模型可能看起来在基于简单指标的改进，但实际上其泛化能力或执行复杂任务的能力正在下降。 AI

影响强调了需要更复杂的评估方法来准确评估 LLM 的能力，并防止误导性的性能指标。

排序理由该条目描述了一项关于 LLM 强化学习中评估指标局限性的研究发现。[lever_c_demoted from research: ic=1 ai=1.0]

AI 生成摘要 · Google Gemini · 来自 1 个来源。我们如何撰写摘要 →

本文如何被排名

Signal score: 42 / 100; Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket: Tool; 该条目描述了一项关于 LLM 强化学习中评估指标局限性的研究发现。[lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration: Single-source cluster; Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics: paper, model release; Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance: High; Clearly on-topic for AI-industry coverage.
Story freshness: Breaking (< 6h); Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

完整方法见我们的编辑标准。

dev.to — LLM tag TIER_1 English(EN) · howcani howcani · 2026-09-05 05:08

所有贪婪的指标都显示模型正在改进。然后 pass@64 从 0.83 下降到 0.19

<p>Every number we watched said the run was working. Correct-per-sample probability tripled. The greedy accuracy curve was climbing. By the numbers on our dashboard, this was a textbook RLVR win.</p> <p>Then we sampled the checkpoint 64 times per problem instead of once. <strong>…