New LEGO benchmark reveals vision-language model limitations in fine-grained understanding

Researchers have introduced LEGO Co-builder, a new benchmark designed to test the fine-grained vision-language understanding capabilities of AI models when interpreting multimodal assembly instructions. The benchmark combines real-world LEGO assembly logic with procedurally generated scenes to evaluate instruction following, object detection, and state detection. While models like InstructBLIP achieved high performance in object detection, advanced models such as GPT-4o and Gemini struggled with fine-grained scene understanding and assembly state detection, highlighting current limitations in these areas. AI

IMPACT Highlights significant gaps in current vision-language models for complex, fine-grained spatial reasoning and state detection, potentially guiding future research.

RANK_REASON The cluster describes a new academic paper introducing a benchmark for evaluating AI models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Gemini
• GLM-4.1-thinking
• GPT-4o
• InstructBLIP
• Jiahuan Pei
• LEGO Co-builder
• MiniGPT-v2
• Qwen VL

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →