OpenAI审计了广泛用于评估AI编码能力的基准SWE-Bench Pro，并发现其不可靠。他们的调查显示，该基准的相当一部分任务存在缺陷，包括问题损坏、隐藏需求和评分标准不完整等问题。因此，OpenAI撤回了其对研究界使用SWE-Bench Pro评估前沿编码性能的建议，并强调随着编码模型的进步，需要更强大、更值得信赖的评估方法。 AI
影响 强调了随着AI编码模型快速改进，需要更强大、更值得信赖的基准。
排序理由 OpenAI公布了关于一个广泛使用的AI基准不可靠性的调查结果。
AI 生成摘要 · Google Gemini · 来自 8 个来源。 我们如何撰写摘要 →
As coding models improve, evals need to become harder, fairer, and more trustworthy. Better benchmarks help the field understand real progress and where the frontier is moving.
To audit SWE-Bench Pro, we used model-based investigator agents alongside independent reviews from five independent experienced software engineers. That helped us examine tasks at scale while keeping expert judgment at the center. https://t.co/3PNbk57uvF
We audited SWE-Bench Pro, one of the most widely used AI coding benchmarks, and found it no longer reliably measures frontier coding capability. We find 30% of SWE-Bench Pro tasks to be broken, and are retracting our previous recommendation that the research community use it as
Our audit of SWE-Bench Pro found that a meaningful share of public tasks contain issues that can distort results. Some correct solutions fail because of hidden requirements, contradictory instructions, overly strict tests, or incomplete grading criteria. https://t.co/s07aJbNks6
To audit SWE-Bench Pro, we used model-based investigator agents alongside independent reviews from five independent experienced software engineers. That helped us examine tasks at scale while keeping expert judgment at the center. https://t.co/z0Rz37Q1L3
As coding models improve, evals need to become harder, fairer, and more trustworthy. Better benchmarks help the field understand real progress and where the frontier is moving.
Our audit of SWE-Bench Pro found that a meaningful share of public tasks contain issues that can distort results. Some correct solutions fail because of hidden requirements, contradictory instructions, overly strict tests, or incomplete grading criteria. https://t.co/OdIghJH0Xk
We audited SWE-Bench Pro, one of the most widely used AI coding benchmarks, and found it no longer reliably measures frontier coding capability. We find the eval to be saturated at a ~70% noise ceiling, and are retracting our previous recommendation that the research community