Vision-language models fail at basic path following tasks

Researchers have identified a significant failure mode in vision-language models (VLMs) related to visual path following. Even advanced VLMs struggle to consistently trace a designated path, frequently switching to nearby, visually similar alternatives. This issue, termed 'local competition,' persists despite efforts like scaling model size, incorporating reasoning capabilities, or providing explicit tracing instructions. The problem extends beyond controlled environments, impacting real-world scenarios such as untangled cables and metro maps. AI

IMPACT Identifies a critical failure in vision-language models for tasks requiring precise visual navigation, potentially impacting robotics and autonomous systems.

RANK_REASON The cluster contains an academic paper detailing a new finding about the limitations of existing models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• arXiv

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →