Foveated MLLMs match human search outcomes but not process

A new research paper titled "Matched Outcomes, Divergent Gaze: How Foveated MLLMs Search Compared to Humans" explores the differences between human visual search and the behavior of multimodal large language models (MLLMs). The study found that while MLLMs can match or even exceed human performance in target detection and efficiency, their search process itself is fundamentally different. The models exhibit low-entropy, large-amplitude scanpaths that are more consistent with themselves than human scanpaths are with each other, suggesting a single-pass architecture rather than a serial one. AI

IMPACT Highlights limitations of current outcome-based metrics for evaluating AI vision systems and suggests a need for process-level evaluation.

RANK_REASON Research paper published on arXiv detailing findings about MLLM behavior. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• COCO-Search18
• Hugging Face
• Matched Outcomes, Divergent Gaze: How Foveated MLLMs Search Compared to Humans
• Mohamed Amine Kerkouri

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →