Researchers have introduced CountVideoBench, a new benchmark designed to evaluate the object counting capabilities and biases of video-language models. Existing benchmarks suffer from skewed ground-truth count distributions, with a median count of five or less, making it difficult to assess performance on higher cardinalities. CountVideoBench features 2,600 synthetic videos with object counts ranging from 5 to 50, controlled for confounds like object motion. Evaluations of sixteen models showed that none exceeded 27.6% Exact Match accuracy, and performance varied significantly across models, revealing a tendency for models to concentrate predictions on specific integers rather than accurately counting. AI
IMPACT Highlights the need for more robust benchmarks to accurately assess and improve the object counting abilities of video-language models.
RANK_REASON The cluster describes a new academic benchmark for evaluating AI models. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
