New MedQA-MM benchmark exposes reasoning shortcuts in medical AI models

Researchers have introduced MedQA-MM, a new benchmark designed to identify "reasoning inflation" in medical multimodal question-answering models. This inflation occurs when models achieve correct answers not through genuine visual reasoning, but by exploiting shortcuts present in text, image annotations, or answer wording. The benchmark's audits and modality ablations reveal that while full-input accuracy can reach 62.63%, text-only and options-only settings significantly reduce this to 53.96% and 29.71% respectively. A curated 1,000-item subset, MedQA-MM, further mitigates these shortcuts, reducing text-only and options-only accuracy to 5.21% and 12.33%, highlighting the need for route-level evidence in medical image-reasoning claims. AI

IMPACT Highlights the need for robust evaluation methods to ensure AI models are not exploiting superficial patterns rather than genuine reasoning.

RANK_REASON The cluster contains a research paper detailing a new benchmark for evaluating AI models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• alphaXiv
• arXiv
• CatalyzeX
• Connected Papers
• CORE Recommender
• DagsHub
• Gotit.pub
• Hugging Face
• Influence Flower
• Litmaps
• MedQA-MM
• ScienceCast
• scite Smart Citations

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →