New VCG-Bench benchmark evaluates vision-language models on structured diagram tasks

Researchers have introduced VCG-Bench, a new benchmark designed to evaluate vision-language models (VLMs) on structured diagram generation and editing tasks. This benchmark utilizes a "Diagram-as-Code" approach with mxGraph XML to enable precise symbolic logic for creating and modifying diagrams, moving beyond traditional pixel-based synthesis. VCG-Bench includes a dataset of 1,449 diagrams across six domains and a tailored evaluation protocol with metrics like Execution Success Rate and Style Consistency Score, highlighting current VLMs' limitations in structured fidelity and reasoning. AI

IMPACT This benchmark could drive improvements in VLM capabilities for structured visual tasks, impacting fields requiring precise diagrammatic representations.

RANK_REASON The cluster describes a new benchmark and dataset for evaluating AI models, which falls under research. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• Hugging Face
• mxGraph
• Peijie Dong
• Style Consistency Score
• VCG-Bench
• vision-language model
• XML

• paper
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →