New EC-Bench reveals multimodal LLMs struggle with long-video counting

A new benchmark called EC-Bench has been developed to evaluate the quantitative reasoning capabilities of multimodal large language models (MLLMs) on long videos. The benchmark includes over 1,600 queries across 152 videos, each longer than 30 minutes, with human-verified evidence spans. Current MLLMs perform poorly on this benchmark, with the best models achieving only around 30% accuracy on enumeration and 24% on counting, significantly below human performance. The research indicates that errors in counting are often linked to difficulties in enumerating relevant instances and grounding evidence temporally, suggesting that video counting is more about evidence retrieval and aggregation than simple arithmetic. AI

IMPACT Highlights significant limitations in current multimodal LLMs for complex video understanding tasks, indicating a need for improved temporal reasoning and evidence aggregation capabilities.

RANK_REASON The cluster contains a research paper introducing a new benchmark for evaluating multimodal LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• EC-Bench
• multimodal large language models
• Taiki Miyanishi

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →