A new paper highlights significant issues with the IntentBench benchmark for evaluating multimodal large language models (MLLMs) in social audio-visual understanding. Researchers found that a substantial portion of IntentBench questions are flawed or can be answered without video input, leading to the release of a refined benchmark called Intentbench-Prime. The study also revealed that current chain-of-thought reasoning methods are costly and surprisingly ineffective, with a simple Vanilla SFT baseline achieving comparable or better results at a fraction of the cost. Furthermore, the analysis suggests that MLLMs can learn significant knowledge from text alone, even outperforming video-based approaches when only textual captions are provided, indicating limitations in current models' social understanding capabilities. AI
IMPACT Highlights limitations in current MLLMs for social understanding and proposes a more reliable benchmark and baseline.
RANK_REASON Research paper detailing benchmark flaws and a new baseline model.
Read on Hugging Face Daily Papers →
AI-generated summary · Google Gemini · from 2 sources. How we write summaries →
