English(EN) What pass@k Cannot Measure: Evaluating Diversity and Capability Retention after Post-Training

AI模型评估指标pass@k因多样性和能力保留问题受到质疑

作者 PulseAugur 编辑部 · [1 个来源] · 2026-10-07 04:00

一项新的研究论文质疑了pass@k指标在评估AI模型方面的有效性，特别是在训练后微调之后。研究表明，虽然pass@k可能显示出边际改进，但它未能捕捉到输出多样性和能力保留等关键方面。使用Qwen2.5-1.5B-Instruct模型在数学问题上的实验显示，不同的微调方法导致多样性度量出现相反的趋势，但pass@k指标在硬问题覆盖率方面并未显示出明显的赢家或显著改进。该论文认为，pass@k可能具有误导性，尤其是在评估正确答案的多样性时，并建议当前的评估协议可能无法准确认证它们旨在衡量的属性。 AI

影响强调了当前AI模型评估中潜在的缺陷，表明需要更强大的指标来捕捉多样性和真实的能力提升。

排序理由分析AI模型评估指标的研究论文。[lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.LG 阅读 →

AI 生成摘要 · Google Gemini · 来自 1 个来源。我们如何撰写摘要 →

本文如何被排名

Signal score: 18 / 100; Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket: Tool; 分析AI模型评估指标的研究论文。[lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration: Single-source cluster; Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics: paper, model release; Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance: High; Clearly on-topic for AI-industry coverage.
Story freshness: Breaking (< 6h); Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

完整方法见我们的编辑标准。

报道来源 [1]

arXiv cs.LG TIER_1 English(EN) · Subham Rath, Raj Dandekar, Rajat Dandekar, Sreedath Panat · 2026-10-07 04:00

pass@k 无法衡量：训练后多样性和能力保留的评估

arXiv:2610.07405v1 Announce Type: new Abstract: pass@$k$, the fraction of problems a model solves within $k$ sampled attempts, is the field's default protocol for deciding whether reinforcement-learning (RL) post-training on verifiable rewards improved a model. At the population …

本文如何被排名

报道来源 [1]

pass@k 无法衡量：训练后多样性和能力保留的评估

相关实体

相关话题