Position: State-of-the-Art Claims Require State-of-the-Art Evidence
A new paper published on arXiv argues that current state-of-the-art claims in AI and machine learning research are often not supported by robust evidence. The authors analyzed ten cross-domain benchmarks and found that in over half of top-model comparisons, the claimed superiority was not consistently demonstrated across tasks or was driven by outlier datasets. They advocate for more precise and honest reporting of benchmark results to accurately reflect the strength of the evidence. AI
IMPACT Highlights potential overstatements in AI benchmark results, urging for more rigorous reporting standards.