VLMs fail to re-examine images when prompted, study finds

Researchers have developed a new framework called VisualSwap to test whether Vision-Language Models (VLMs) truly re-examine images when they claim to. Their experiments using the VS-Bench dataset on models like Qwen3-VL and Kimi-VL showed that these models frequently fail to detect semantic changes in images, even when visually similar. This suggests that VLMs often generate text about visual re-examination without actually performing it, a tendency exacerbated in models designed for more complex reasoning. AI

IMPACT Challenges the perceived visual understanding of current VLMs, suggesting a need for improved grounding mechanisms beyond textual cues.

RANK_REASON Academic paper introducing a new framework and dataset for evaluating VLM capabilities. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• ERNIE-VL
• Kimi-VL
• Qwen3-VL
• VisualSwap
• VS-Bench

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →