Multimodal LLMs evaluated on calligraphy quality assessment

A new research paper explores the capabilities of multimodal large language models in evaluating the quality of calligraphic brushstrokes and providing educational feedback. The study tested GPT-4o, Claude Sonnet 4, and Gemini 2.5 Flash, comparing their assessments against those of human experts. While the models demonstrated useful accuracy in scoring, they did not consistently correlate with expert rankings, and analysis revealed distinct evaluative biases in each model. AI

IMPACT This research highlights the current limitations of LLMs in nuanced qualitative assessment and educational feedback, suggesting areas for future development in AI's role in creative fields.

RANK_REASON The cluster contains a research paper published on arXiv detailing an evaluation of LLM capabilities. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• arXiv
• Claude
• Claude Sonnet 4
• Gemini 2.5 Flash
• GPT-4o
• retrieval-augmented generation

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →