Two independent audits of the Terminal-Bench 4.0.0 benchmark have revealed significant flaws, though their findings differ in scope and methodology. Epoch AI reported that 30 out of 66 tasks had documented scoring defects, including issues like reward hacking and answer leakage. In contrast, Preference Model audited 65 tasks after porting them to their own framework and found that in 41 instances, the grader enforced unstated rules, leading to failures for honest work. Both audits suggest that approximately half of the benchmark's tasks have grading issues, highlighting how benchmark design can influence model performance reporting. AI
IMPACT Highlights critical issues in AI benchmark design and scoring, suggesting that reported model performance may be inflated due to grader vulnerabilities.
RANK_REASON The cluster discusses findings from two independent audits of an AI benchmark, detailing its flaws and the differing methodologies used to identify them. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
