Fact-checking benchmarks risk contamination, new study finds

A new research paper questions the effectiveness of dynamic evaluation methods for multimodal automated fact-checking (MAFC). The study reveals that even benchmarks designed with claims published after an LLM's knowledge cut-off date still suffer from contamination, where claims can be verified using pre-existing knowledge. This contamination can inflate performance metrics by up to 11.34 points and alter the perceived rankings of different MAFC systems. The researchers propose practical guidelines for more trustworthy MAFC evaluations. AI

IMPACT Highlights potential overestimation of AI fact-checking capabilities and suggests improvements for reliable evaluation.

RANK_REASON The cluster contains a research paper detailing new findings and methodologies for evaluating AI systems. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• alphaXiv
• arXiv
• AVeriTeC
• CatalyzeX
• ClaimReview2025Q4
• DagsHub
• Gotit.pub
• Hugging Face
• ScienceCast

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →