New benchmark reveals multimodal LLMs struggle with video OCR

A new benchmark called MME-VideoOCR has been introduced to evaluate the optical character recognition (OCR) capabilities of multimodal large language models (MLLMs) specifically within video content. The benchmark highlights that current MLLMs struggle with video OCR due to motion blur and temporal variations, with even the top-performing model, Gemini 2.5 Pro, achieving only 73.7% accuracy. The research indicates that MLLMs perform better on tasks requiring single-frame text recognition but falter when holistic video comprehension, spatio-temporal reasoning, or cross-frame information integration is needed. AI

IMPACT Highlights limitations in current multimodal LLMs for video OCR, suggesting areas for future model development and training.

RANK_REASON The cluster contains a research paper introducing a new benchmark for evaluating multimodal LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• Gemini 2.5 Pro
• MME-VideoOCR
• Multimodal Large Language Models and Tunings: Vision, Language, Sensors, Audio, and Beyond
• optical character recognition
• Yang Shi

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →