A new benchmark called ANCHOR has been introduced to evaluate the grounded reasoning capabilities of multimodal large language models (MLLMs). ANCHOR comprises over 2,300 videos and 8,000 question-answer pairs, with a focus on assessing whether models' explanations align with relevant perceptual evidence, such as visual regions, temporal intervals, and audio cues. Initial testing across twelve MLLMs revealed a significant gap between answer correctness and the accuracy of their stated reasoning, indicating that current evaluation methods may obscure important grounding failures. The benchmark also demonstrated that removing video input drastically reduces accuracy, while removing bounding boxes or audio has a smaller, model-dependent impact. AI
IMPACT Highlights the need for more robust evaluation of multimodal AI reasoning beyond simple answer accuracy.
RANK_REASON The item is an academic paper introducing a new benchmark for evaluating AI models. [lever_c_demoted from research: ic=1 ai=1.0]
- alphaXiv
- arXiv
- CatalyzeX
- Connected Papers
- DagsHub
- Gemini
- generative pre-trained transformer
- Gotit.pub
- Hugging Face
- Litmaps
- Multimodal Large Language Models and Tunings: Vision, Language, Sensors, Audio, and Beyond
- ScienceCast
- scite Smart Citations
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
