New research reveals instability in short-answer VQA benchmarks

A new paper published on arXiv highlights significant instability in short-answer visual question answering (VQA) benchmarks. The research indicates that current benchmarks often conflate the semantic correctness of a model's answer with its surface-form match to an expected response. This instability is particularly pronounced in text-rich benchmarks, where up to half of reported errors are semantically acceptable answers penalized solely for format mismatch. The study also found that minor changes in prompts or context can substantially alter benchmark outcomes, suggesting that official VQA scores should include semantic audits and answer-type diagnostics for better interpretability. AI

RANK_REASON The item is a research paper published on arXiv detailing findings about the instability of benchmarks. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• alphaXiv
• arXiv
• benchmark
• CatalyzeX Code Finder for Papers
• central processing unit
• Connected Papers
• DagsHub
• Gotit.pub
• Hugging Face
• Influence Flower
• Litmaps
• ScienceCast
• scite Smart Citations
• semantic judge
• Vision--Language Models
• visual question answering

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →