A new study published on arXiv evaluates the performance of large language models (LLMs) in annotating the severity of mental health crises, specifically focusing on veteran-related social media content. The research compared LLM annotations against those from eight Veteran Subject Matter Experts (SMEs), finding that while some LLMs demonstrated higher inter-model consistency than human experts, this did not always translate to agreement with human judgments or reliable detection of rare, high-severity cases. Proprietary models generally performed better and were more stable than open-source alternatives, with GPT-5.6-Terra showing the best overall performance and Qwen2.5-7B excelling at identifying acute suicidal ideation. AI
IMPACT LLMs show promise for scaling mental health annotation but require careful validation for high-stakes applications.
RANK_REASON The cluster contains an academic paper detailing research findings on LLM performance in a specific domain. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
