A new research paper published on arXiv evaluates the performance of 12 Vision-Language Models (VLMs) on recognizing Arabic calligraphy, a culturally significant visual domain. The study found that even the best-performing models struggled, with top accuracy reaching only 60.9% for closed-API models and 47.2% for open-weight models. A significant issue identified was "default-label collapse," where models disproportionately predicted a single calligraphy style, such as Thuluth, regardless of the actual image content. Prompting strategies showed mixed results, sometimes improving accuracy but often shifting the default prediction rather than eliminating it, highlighting robustness issues in current VLMs. AI
IMPACT Highlights limitations in VLM robustness and cultural understanding, suggesting a need for more diverse and culturally sensitive evaluation datasets.
RANK_REASON Academic paper evaluating AI model capabilities on a specific task. [lever_c_demoted from research: ic=1 ai=1.0]
- Arabic calligraphy
- arXiv
- Diwani
- GPT-4o
- Hugging Face
- Niaz Mohaiman Abtahi
- Thuluth
- Vision--Language Models
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
