New benchmark NeMo challenges video LLMs on temporal understanding

Researchers have introduced NeMo, a novel task and benchmark called NeMoBench, designed to evaluate the temporal understanding capabilities of video large language models (VideoLLMs). The task, inspired by the 'needle in a haystack' test, focuses on retrieval-style long-context recall and temporal grounding. NeMoBench comprises over 31,000 question-answer pairs derived from thousands of videos, with a scalable automated pipeline ensuring its continuous updateability. Experiments on 20 state-of-the-art models reveal their current strengths and weaknesses in temporal understanding. AI

IMPACT Introduces a new benchmark to push the capabilities of video large language models in temporal understanding.

RANK_REASON The cluster describes a new academic paper introducing a novel benchmark and task for evaluating AI models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• alphaXiv
• arXiv
• CatalyzeX
• DagsHub
• Gotit.pub
• Hugging Face
• NeMo
• NeMoBench
• ScienceCast
• VideoLLMs
• Zi-Yuan Hu

• paper
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →