Video-HOCA benchmark reveals Video-LLMs struggle with anomaly explanation

A new diagnostic benchmark called Video-HOCA has been introduced to evaluate the physical anomaly reasoning capabilities of Video Large Language Models (Video-LLMs). This benchmark utilizes an Ontological-Causal taxonomy and comprises over 1,400 videos with more than 3,470 question-answer pairs, verified by humans. Initial testing on 20 Instruct-mode Video-LLMs revealed that while recognition tasks scored around 75-88%, explanation tasks (Task II) had macro-F1 scores mostly below 50%, indicating a significant gap between recognizing anomalies and explaining them. AI

IMPACT Highlights a key limitation in current Video-LLMs, suggesting a need for improved reasoning and explanation capabilities beyond simple recognition.

RANK_REASON The cluster describes a new academic paper introducing a benchmark for evaluating AI models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• arXiv
• Chang Liu
• Hugging Face
• Instruct-mode Video-LLMs
• Ontological-Causal
• Video-HOCA
• Video LLMs

• paper
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →