AI chatbots excel at emergency psychiatric triage but over-assign urgency

A new study evaluated 15 advanced AI chatbots on their ability to perform emergency psychiatric triage using 112 clinical vignettes. The chatbots demonstrated high accuracy in identifying true emergencies, with an under-triage rate of only 5.6% for critical cases. However, they exhibited a tendency to over-triage less urgent situations, resulting in a mean signed ordinal error indicating a net over-estimation of urgency. Overall accuracy across all triage levels varied significantly between models, ranging from 42.0% to 71.8%. AI

IMPACT Highlights the need for improved calibration in AI models used for sensitive applications like mental health triage.

RANK_REASON Academic paper evaluating AI chatbot performance on a specific task.

Read on arXiv cs.AI →

• AGIEval
• arXiv
• Claude 3 Opus
• NeurIPS
• Gemini 1.5 Pro
• GPT-4
• Llama 3
• MMLU
• Mistral Large
• Veith Weilnhammer
• AI chatbots

• paper
• safety
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →