PulseAugur
EN
LIVE 06:28:08

AI code benchmarks lack rigor, new papers reveal flaws and propose solutions

By PulseAugur Editorial · [2 sources] ·

Two new research papers highlight critical issues in evaluating large language models (LLMs) on code-related tasks. The first paper, "Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility," surveyed 672 code benchmarks and found a significant lag between awareness of benchmark quality and actual practice, introducing guidelines to improve rigor. The second paper, "Is Your Benchmark Still Useful? Dynamic Benchmarking for Code Language Models," proposes a dynamic benchmarking framework to address data contamination by transforming inputs, revealing that current models perform worse and rankings shift dramatically on these dynamic benchmarks. AI

IMPACT Highlights critical flaws in current AI code evaluation methods, suggesting a need for more robust and dynamic benchmarking to accurately assess model capabilities and prevent data contamination.

RANK_REASON Two academic papers published on arXiv proposing new methodologies and guidelines for code benchmarks.

Read on arXiv cs.AI →

AI-generated summary · Google Gemini · from 2 sources. How we write summaries →

AI code benchmarks lack rigor, new papers reveal flaws and propose solutions

COVERAGE [2]

  1. arXiv cs.AI TIER_1 English(EN) · Jialun Cao, Yuk-Kit Chan, Zixuan Ling, Wenxuan Wang, Shuqing Li, Mingwei Liu, Ruixi Qiao, Yuting Han, Chaozheng Wang, Boxi Yu, Pinjia He, Shuai Wang, Zibin Zheng, Michael R. Lyu, Shing-Chi Cheung ·

    Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility

    arXiv:2501.10711v5 Announce Type: replace-cross Abstract: Code-related benchmarks play a critical role in evaluating large language models (LLMs), yet their quality fundamentally shapes how the community interprets model capabilities. In the past few years, awareness of benchmark…

  2. arXiv cs.CL TIER_1 English(EN) · Batu Guan, Xiao Wu, Yuanyuan Yuan, Shaohua Li ·

    Is Your Benchmark Still Useful? Dynamic Benchmarking for Code Language Models

    arXiv:2503.06643v2 Announce Type: replace-cross Abstract: In this paper, we tackle a critical challenge in model evaluation: how to keep code benchmarks useful when models might have already seen them during training. We introduce a novel solution, dynamic benchmarking framework,…

RELATED ENTITIES

RELATED TOPICS