New benchmark reveals LLMs struggle with clinical diagnostic reasoning

Researchers have introduced SUP-MIMIC, a new benchmark designed to evaluate the robustness of large language models (LLMs) in clinical diagnosis. This framework, built upon the MIMIC-IV-v3.1 dataset, includes tasks that test LLMs' ability to handle diagnostic ambiguity and identify common diseases from varied symptoms. Initial evaluations show that current state-of-the-art LLMs struggle with these tasks, relying on statistical shortcuts rather than genuine causal reasoning, which could lead to missed diagnoses in real-world medical applications. AI

IMPACT Highlights critical safety concerns for LLMs in clinical settings, potentially slowing adoption until reasoning capabilities improve.

RANK_REASON The item describes a new academic paper introducing a benchmark for evaluating LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• Diagnostic Convergence Task
• Diagnostic Divergence Task
• LLMs
• SUP-MIMIC

• paper
• safety
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →