New benchmark CRAG-MM-Diagnostics reveals VLM bottlenecks in KI-VQA

Researchers have introduced CRAG-MM-Diagnostics, a new benchmark designed to analyze the performance of Vision-Language Models (VLMs) in Knowledge-Intensive Visual Question Answering (KI-VQA). This benchmark provides stage-wise annotations to pinpoint failures in areas like visual grounding, object identification, and knowledge retrieval. Initial evaluations reveal that knowledge retrieval and reasoning are significant bottlenecks for current VLMs, though issues also exist in object identification and image retrieval integration. The study also proposes a new retrieval-augmented pipeline that improved the accuracy of GPT-5 and Qwen models. AI

IMPACT This benchmark could lead to more robust and accurate vision-language models by identifying specific areas for improvement in knowledge retrieval and reasoning.

RANK_REASON The cluster contains a research paper detailing a new benchmark and evaluation of AI models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• CRAG-MM-Diagnostics
• GPT-5
• Hugging Face
• KI-VQA
• Qwen
• vision-language model

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →