English(EN) Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility

AI代码基准测试缺乏严谨性，新论文揭示缺陷并提出解决方案

作者 PulseAugur 编辑部 · [2 个来源] · 2026-07-07 04:00

两篇新研究论文强调了在代码相关任务上评估大型语言模型（LLMs）时存在的关键问题。第一篇论文《代码基准测试应优先考虑严谨性、可靠性和可复现性》调查了672个代码基准测试，发现对基准测试质量的认识与实际操作之间存在显著差距，并提出了改进严谨性的指导方针。第二篇论文《您的基准测试还有用吗？用于代码语言模型的动态基准测试》提出了一个动态基准测试框架，通过转换输入来解决数据污染问题，结果显示当前模型在这些动态基准测试上的表现更差，排名也发生巨大变化。 AI

影响强调了当前AI代码评估方法中的关键缺陷，表明需要更强大和动态的基准测试来准确评估模型能力并防止数据污染。

排序理由两篇在arXiv上发表的学术论文，提出了代码基准测试的新方法和指导方针。

在 arXiv cs.AI 阅读 →

AI 生成摘要 · Google Gemini · 来自 2 个来源。我们如何撰写摘要 →

报道来源 [2]

arXiv cs.AI TIER_1 English(EN) · Jialun Cao, Yuk-Kit Chan, Zixuan Ling, Wenxuan Wang, Shuqing Li, Mingwei Liu, Ruixi Qiao, Yuting Han, Chaozheng Wang, Boxi Yu, Pinjia He, Shuai Wang, Zibin Zheng, Michael R. Lyu, Shing-Chi Cheung · 2026-07-07 04:00

Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility

arXiv:2501.10711v5 Announce Type: replace-cross Abstract: Code-related benchmarks play a critical role in evaluating large language models (LLMs), yet their quality fundamentally shapes how the community interprets model capabilities. In the past few years, awareness of benchmark…
arXiv cs.CL TIER_1 English(EN) · Batu Guan, Xiao Wu, Yuanyuan Yuan, Shaohua Li · 2026-07-07 04:00

Is Your Benchmark Still Useful? Dynamic Benchmarking for Code Language Models

arXiv:2503.06643v2 Announce Type: replace-cross Abstract: In this paper, we tackle a critical challenge in model evaluation: how to keep code benchmarks useful when models might have already seen them during training. We introduce a novel solution, dynamic benchmarking framework,…

报道来源 [2]

Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility

Is Your Benchmark Still Useful? Dynamic Benchmarking for Code Language Models

相关实体

相关话题