Researchers have introduced EvaOmni, a new benchmark designed to evaluate the audio-visual reasoning capabilities of omnimodal large language models. This benchmark requires models to not only answer questions but also identify and localize the specific audio and visual evidence supporting their answers, addressing limitations in existing benchmarks that may allow for shortcutting. EvaOmni includes 1,027 human-annotated question-answer pairs across 392 videos, covering 15 distinct capabilities. In evaluations, Gemini3.7-Flash demonstrated the highest overall performance, achieving 79.4% accuracy, but its scores on evidence grounding and temporal localization indicate room for improvement in advanced models. AI
IMPACT This benchmark will drive development of more robust audio-visual reasoning capabilities in multimodal AI systems.
RANK_REASON The cluster describes a new academic benchmark for evaluating AI models.
Read on Hugging Face Daily Papers →
AI-generated summary · Google Gemini · from 2 sources. How we write summaries →