VLMs over-correct math OCR, hiding student errors; new metric PINK improves evaluation

Researchers have identified a significant issue in evaluating handwritten math OCR systems, particularly with Vision-Language Models (VLMs). These models often over-correct student errors instead of accurately transcribing them, masking learning opportunities. To address this, a new semantic evaluation metric called PINK has been developed, which uses LLMs to grade and penalize such over-correction. Evaluations on the FERMAT dataset showed that PINK significantly alters model rankings compared to traditional metrics like BLEU, with Gemini 2.5 Flash performing better in faithful transcription. AI

IMPACT Introduces a more accurate evaluation metric for educational AI, potentially influencing future VLM development for math transcription.

RANK_REASON Academic paper introducing a new evaluation metric for a specific AI capability.

Read on arXiv cs.CV →

• arXiv
• BLEU
• Gemini 2.5 Flash
• GPT-4o
• LLM
• PINK

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →