A new open-source benchmark called OpenMTB-Audit has been developed to evaluate the safety of large language models (LLMs) in molecular tumor board (MTB) workflows. The benchmark, comprising 500 synthetic non-small cell lung cancer cases, reveals that current LLMs exhibit significant over-refusal, failing to correctly identify partially supported treatment recommendations. To address this, a framework named MTB-AuditAgent was created, which reduces over-refusal to 6.7% and achieves 91.2% accuracy in safety classification. AI
IMPACT Highlights critical safety limitations in LLMs for medical applications, necessitating specialized frameworks for reliable clinical decision support.
RANK_REASON The cluster contains an academic paper detailing a new benchmark and framework for evaluating LLM safety in a specific domain. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
