Vision-Language Models Show Subtle Hallucinations in Historical Document OCR

A new research paper analyzes the performance of vision-language models (VLMs) in transcribing historical documents, finding that while they outperform traditional optical character recognition (OCR) systems on standard metrics like Character Error Rate (CER) and Word Error Rate (WER), they exhibit subtle but critical failure modes. These include generating spurious content and making semantic substitutions that alter meaning without significantly impacting CER/WER, particularly affecting named entities. The study highlights the need for evaluation methods that assess semantic reliability beyond simple character accuracy for archival transcription. AI

IMPACT Highlights critical semantic reliability gaps in VLMs for historical document transcription, necessitating new evaluation frameworks.

RANK_REASON Research paper published on arXiv detailing limitations of vision-language models for OCR. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• Berrutti dataset
• Character Error Rate
• Diego Belzarena
• optical character recognition
• vision-language model
• word error rate

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →