A new study analyzing over 127,000 queries from 6,342 healthcare professionals reveals a significant divergence between how large language models (LLMs) are evaluated and their actual clinical use. The research found that common tasks like documentation and knowledge retrieval constituted the majority of queries, while diagnostic tasks were infrequent. Furthermore, over a third of these real-world queries could not be adequately answered as posed. When applying the study's framework to 58 existing benchmarks, it was found that these evaluations shared only 31% of the task mix of actual clinical use, indicating that current benchmark scores may not accurately reflect LLM performance in real healthcare settings. AI
IMPACT Current LLM benchmarks may not accurately reflect their utility in clinical settings, necessitating a shift towards evaluations that mirror real-world healthcare tasks.
RANK_REASON The cluster contains an academic paper detailing a new study and analysis. [lever_c_demoted from research: ic=1 ai=1.0]
Read on Hugging Face Daily Papers →
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
