New MTDiag dataset evaluates LLMs in multi-turn clinical diagnosis

Researchers have developed MTDiag, a new multi-turn diagnostic dialogue dataset designed to better evaluate Large Language Models (LLMs) in clinical settings. Unlike static benchmarks, MTDiag simulates the interactive and incremental nature of medical diagnosis by drawing data from DDXPlus, MIMIC-IV, and AJCR case reports. The dataset is normalized using medical knowledge bases like UMLS and ICD-10, and includes a pipeline for generating natural-language utterances. This initiative aims to assess LLMs' capabilities as diagnostic agents beyond simple accuracy, introducing clinically grounded metrics for evaluating multi-turn differential diagnosis. AI

IMPACT This dataset and its associated metrics could lead to more robust LLM evaluations in healthcare, improving their reliability for diagnostic tasks.

RANK_REASON The cluster describes a new dataset and methodology for evaluating LLMs in a specific research domain (clinical diagnosis), presented in an academic paper. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• AJCR
• Alexander Fichtl M.Sc.
• DDXPlus
• ICD-10
• LLMs
• MIMIC-IV
• MTDiag
• UMLS
• UserLM-8B

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →