New benchmark tests if vision-language models ground answers in pixels

Researchers have introduced PriVE-Bench, a new benchmark designed to evaluate how well vision-language models (VLMs) ground their answers in visual evidence rather than relying on learned language or category priors. The benchmark uses paired original and counterfactual images to test if models can distinguish visual reality from common knowledge. Additionally, PriVE-Tools extends this by assessing whether agentic vision systems, using tools like bounding boxes and crops, can improve grounding against these counterfactual conflicts. Initial results indicate that while visual evidence tools can help some models, they do not universally prevent reliance on priors. AI

IMPACT This research could lead to more robust vision-language models that are less susceptible to biases and rely more on actual visual input.

RANK_REASON The cluster describes a new benchmark and tools for evaluating AI models, presented in an academic paper. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• alphaXiv
• arXiv
• CatalyzeX
• DagsHub
• Gotit.pub
• Hugging Face
• PriVE-Bench
• PriVE-Tools
• ScienceCast
• Vision--Language Models

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →