New Chartography Benchmark Reveals AI Struggles with Professional Chart Understanding

A new benchmark called Chartography has been developed to assess professional chart understanding across various domains like medicine, engineering, finance, and science. Unlike existing benchmarks, Chartography features 100 tasks with charts in specialized formats and questions crafted by domain experts, with independent verification. Current frontier models struggle with this benchmark, achieving a maximum of 45.0% accuracy, indicating significant challenges in visual perception, reading nuanced features, and interpreting domain-specific conventions. AI

IMPACT Highlights limitations in current AI models' ability to interpret complex, domain-specific visual data, suggesting a need for improved perception and reasoning capabilities.

RANK_REASON The cluster describes a new academic benchmark for AI evaluation. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• alphaXiv
• arXiv
• CatalyzeX
• Chartography
• DagsHub
• engineering
• finance
• Gotit.pub
• Hugging Face
• industrial manufacturing
• Influence Flower
• medicine
• science
• ScienceCast

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →