When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answering
Researchers have developed OGCaReBench, a new benchmark designed to evaluate how well large language models can answer complex clinical questions that fall outside standard medical guidelines. The benchmark, derived from medical case reports and validated by experts, focuses on free-form, retrieval-based reasoning for rare scenarios. Experiments showed that even advanced models like GPT-5.2 struggled, but augmenting them with retrieved medical articles significantly improved performance, highlighting the need for evidence-grounding in medical AI. AI
IMPACT This benchmark will drive the development of LLMs capable of handling complex, real-world medical scenarios, improving AI's utility in clinical decision support.