Researchers have developed a new method to evaluate multimodal large language models (LLMs) by separating their visual perception from their counting abilities. By presenting models with diagrams and asking questions about both simple and complex counts within them, the study aims to pinpoint where failures occur. Tests on models like Gemini, Gemma, and Qwen3-VL revealed that while these models can often read small diagrams accurately, they struggle with complex counting tasks, with performance degrading as the answer size increases rather than its complexity class. When provided with code and transcriptions, some models demonstrated improved counting accuracy, suggesting that reading comprehension can become a bottleneck. AI
IMPACT This research introduces a novel method to better understand and improve the capabilities of multimodal LLMs in processing visual information and performing complex reasoning tasks.
RANK_REASON Research paper published on arXiv detailing a new evaluation method for multimodal LLMs. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
