AI models can answer video questions without watching videos, study finds

A new audit of four public benchmarks for traffic accident Video Question Answering (VideoQA) reveals that several open-weight Vision-Language Models (VLMs) can achieve competitive accuracy without using visual evidence, relying instead on textual shortcuts. In some cases, removing video input actually improved accuracy, and adding more frames degraded performance. To address this, researchers introduced metrics like Blind Gap and Visual Gain to quantify visual dependence and a Shortcut Score to filter out questions prone to textual shortcuts, aiming to improve visual grounding in safety-critical applications. AI

IMPACT Highlights the need for robust evaluation methods to ensure AI models genuinely utilize multimodal inputs, especially in safety-critical applications.

RANK_REASON Research paper detailing a new audit methodology and findings for evaluating AI models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on Hugging Face Daily Papers →

• Accuracy
• Hugging Face
• Modality Collapse
• Traffic VideoQA
• Vision-Language Models
• Visual Dependence and Multisensory Balance Rehabilitation After Stroke

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →