A new paper introduces the Visual Dependency Gap (VDG) to assess the visual grounding capabilities of video large language models (LLMs). The VDG measures the difference in accuracy between models processing original video and black screens, revealing that many models' performance gains are not due to visual understanding but rather frame diversity. The study found that temporal reasoning contributes little to accuracy, and even stable aggregate accuracy can mask significant question-level answer flips. The VDG is proposed as a standard audit for evaluating visually grounded capabilities in video LLMs. AI
IMPACT This research highlights a critical flaw in current video LLM benchmarks, suggesting a need for more robust evaluation methods that truly measure visual understanding.
RANK_REASON Academic paper introducing a new diagnostic metric for evaluating video LLMs. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
