Vision-language models rely on ViT position codes for spatial understanding

A new research paper investigates how vision-language models interpret spatial relationships, specifically left-right distinctions. The study found that the position codes within the vision transformer (ViT) are crucial for these models to correctly identify left-right orientations. Experiments with models like Qwen2.5-VL and InternVL3 demonstrated that altering these ViT position codes significantly impacts their ability to answer left-right questions accurately, while changes to the language model's token order had a lesser effect. AI

IMPACT This research highlights the importance of specific architectural components like ViT position codes for accurate spatial reasoning in vision-language models.

RANK_REASON Research paper analyzing model capabilities. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• CV-Bench
• InternVL3
• InternVL3-2B
• Qwen2.5-VL
• Qwen2.5-VL-3B
• Qwen2.5-VL-7B
• Qwen3-VL
• Qwen3-VL-4B

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →