New VidNum-1.4K benchmark reveals VLM struggles with video numerical reasoning

Researchers have introduced VidNum-1.4K, a new benchmark designed to test the numerical reasoning capabilities of vision-language models (VLMs) using video data. This benchmark features 1,379 video-question pairs that require multi-step numerical logic, arithmetic operations, and comparisons grounded in temporal evidence. Evaluations showed that current state-of-the-art VLMs, including Gemini-3.1 Pro, struggle with this task, highlighting a significant gap in their ability to develop robust internal world models for video comprehension. AI

IMPACT Highlights a critical gap in current VLMs' ability to perform complex numerical reasoning in video, indicating a need for improved internal world models.

RANK_REASON The item is a research paper introducing a new benchmark for evaluating AI models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• Gemini-3.1 Pro
• Shaoyang Cui
• VidNum-1.4K
• vision-language model

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →