New benchmark reveals LLMs struggle with patient triage accuracy

A new benchmark called CARE-Bench has been developed to evaluate the safety and effectiveness of large language models (LLMs) in patient-facing medical triage scenarios. The benchmark, which includes 500 cases and over 1,000 evaluated dialogue prefixes, assesses how well models recommend the next appropriate action for a patient. Evaluations of 11 different LLMs showed that even with prompting, models struggled with accuracy, often recommending care prematurely or failing to gather necessary clarifying information. AI

IMPACT Highlights critical safety concerns for LLMs in healthcare, suggesting current models are not ready for unsupervised patient triage.

RANK_REASON The cluster describes a new academic paper introducing a benchmark for evaluating LLM performance. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• CARE-Bench
• GPT-5.5
• LLM

• paper
• safety
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →