English(EN) What Does a Harness Repair? A Preregistered Study of Visibility, Baseline Adequacy and Evaluation Defects

AI评估工具的缺陷与有效性研究

作者 PulseAugur 编辑部 · [1 个来源] · 2026-10-27 04:00

一项新发表在arXiv上的预注册研究，调查了AI模型评估工具的有效性，特别关注可见性、基线充分性和缺陷。研究使用了从1.7B到8B参数的模型，包括Qwen3-14B，在LongevityBench、MMLU-Pro和GSM8K等基准测试中进行。研究结果表明，禁用模型中某些“思考”机制可以通过允许解析器处理以前无法读取的答案来提高准确性，并且对于较小的模型，这种方法与救援配置或GEPA选择的评估工具相当。 AI

影响这项研究突显了AI评估方法中潜在的缺陷，并提出了改进建议以实现更准确的基准测试。

排序理由该集群包含一篇发表在arXiv上的研究论文，详细介绍了对AI模型评估的研究。[lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.AI 阅读 →

AI 生成摘要 · Google Gemini · 来自 1 个来源。我们如何撰写摘要 →

本文如何被排名

Signal score: 10 / 100; Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket: Tool; 该集群包含一篇发表在arXiv上的研究论文，详细介绍了对AI模型评估的研究。[lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration: Single-source cluster; Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics: paper, model release; Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance: High; Clearly on-topic for AI-industry coverage.
Story freshness: Same-day; Cluster formed today. Ranking reflects the current source set at time of score.

完整方法见我们的编辑标准。

报道来源 [1]

arXiv cs.AI TIER_1 English(EN) · Bowen Xu, Boyu Chen · 2026-10-27 04:00

什么会修复线束？一项关于可见性、基线充分性和评估缺陷的预注册研究

arXiv:2610.05533v2 Announce Type: replace-cross Abstract: Harness search keeps a change to the prompts, token budgets or parsers around a frozen model if the change raises a score. Such a gain can come from answers the parser could not read before, a weak comparison or an evaluat…

本文如何被排名

报道来源 [1]

什么会修复线束？一项关于可见性、基线充分性和评估缺陷的预注册研究

相关实体

相关话题