New MMR-V benchmark reveals LLMs struggle with deep video reasoning

A new benchmark called MMR-V has been introduced to evaluate the multimodal deep reasoning capabilities of large language models (LLMs) when processing video content. Unlike existing benchmarks that focus on simple frame matching, MMR-V requires models to perform long-range, multi-frame reasoning and infer information beyond direct perception. Experiments using MMR-V, which comprises 1,257 tasks across 317 videos, revealed that even advanced models like Gemini 2.5 Pro struggle, achieving only 64.3% accuracy, and common reasoning enhancement strategies like Chain-of-Thought offer limited improvements. AI

IMPACT Highlights limitations in current LLMs for complex video understanding, potentially guiding future research in multimodal reasoning.

RANK_REASON The item is a research paper introducing a new benchmark for evaluating AI models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• alphaXiv
• arXiv
• CatalyzeX
• Chain-of-Thought
• DagsHub
• Gemini 2.5 Pro
• Gotit.pub
• Hugging Face
• Kejian Zhu
• Multimodal Large Language Models
• ScienceCast

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →