New benchmark RxScribe Bench evaluates vision-language models on prescription accuracy

Researchers have introduced RxScribe Bench, a new benchmark designed to evaluate vision-language models on their ability to transcribe handwritten Indian outpatient prescriptions. Unlike previous methods that aggregate performance into a single score, RxScribe Bench decomposes evaluation into four distinct axes: Correctness, Hallucination, Engagement, and Robustness. This multi-axis approach aims to better reflect the varying clinical risks associated with different types of transcription errors. Initial evaluations on frontier vision-language models revealed that no single model excelled across all four axes, highlighting the complexity of prescription digitization. AI

IMPACT This benchmark could lead to more reliable vision-language models for healthcare applications by focusing on critical error types.

RANK_REASON The item describes a new benchmark for evaluating AI models, presented in an academic paper on arXiv. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• arXiv
• correctness
• hallucination
• Indian
• robustness
• RxScribe Bench
• social engagement

• paper
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →