Researchers have introduced OmniCompass, a new benchmark designed to holistically evaluate the video understanding capabilities of multimodal large language models (MLLMs). The benchmark includes two tracks, OmniCompass-VL for vision-language tasks and OmniCompass-AV for audio-visual tasks, each featuring temporal grounding, spatial grounding, and question answering. Evaluations of 50 MLLMs revealed that while proprietary models generally outperform open-source counterparts, precise spatiotemporal grounding and audio-visual reasoning remain significant challenges for even the leading models. AI
IMPACT This benchmark aims to provide a more comprehensive evaluation of MLLMs' video understanding, highlighting areas for improvement in grounding and audio-visual reasoning.
RANK_REASON The cluster describes a new academic paper introducing a benchmark for evaluating AI models. [lever_c_demoted from research: ic=1 ai=1.0]
Read on Hugging Face Daily Papers →
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
