New MIRA-Bench reveals vision-language models struggle with mirror reflections

Researchers have developed MIRA-Bench, a new benchmark designed to test the spatial reasoning capabilities of vision-language models (VLMs) in scenarios involving mirrors. Existing benchmarks often overlook the complexities introduced by reflections, where depth data can be distorted while visual appearance remains intact. MIRA-Bench includes over 1,000 RGB-D images from diverse real-world scenes, annotated with reflective surface masks and object reflections. Evaluations on state-of-the-art VLMs revealed significant failures in identifying reflections, matching objects to their real counterparts, and making accurate spatial inferences, with depth data even degrading performance. AI

IMPACT Highlights a critical gap in VLM spatial reasoning, potentially impacting robotics and augmented reality applications.

RANK_REASON The cluster describes a new benchmark and dataset for evaluating AI models, presented in an academic paper. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• arXiv
• CatalyzeX
• DagsHub
• Gotit.pub
• Hugging Face
• MIRA-Bench
• RGB-D
• vision-language model

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →