LVLMs struggle with diagram relationships despite internal signals

Researchers have identified a disconnect between the internal representations and the predictive capabilities of large vision-language models (LVLMs) when processing diagrams. Despite LVLMs showing proficiency in diagram understanding benchmarks, they often fail to grasp relationships, particularly those involving directed edges. Through experiments with synthetic diagrams and flowcharts, the study found that signals related to nodes and edges are decodable from the models' hidden states, and in some cases, these signals causally influence the model's predictions. This suggests a gap between the information the models process internally and the outputs they generate. AI

IMPACT Reveals limitations in current LVLM diagram understanding, suggesting areas for future model development.

RANK_REASON Academic paper detailing research findings on LVLM capabilities. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• Haruto Yoshida
• Hugging Face
• Large vision-language models

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →