New benchmark reveals LLMs overestimate medical competence

Researchers have developed a new benchmark for evaluating large language models (LLMs) on Polish medical exams, expanding the dataset with over 15,000 questions and structural modifications to better assess true competence beyond simple multiple-choice guessing. Their study found that the best-performing model, Qwen3.5-122B, saw a significant drop in scores under the new, more challenging evaluation setup. The findings suggest that current LLM evaluations in medicine may overestimate their capabilities due to biases and test design, and the new benchmark is being made publicly available. AI

IMPACT New evaluation methods challenge current LLM performance metrics, suggesting a need for more robust testing in specialized domains like medicine.

RANK_REASON The cluster contains an academic paper introducing a new benchmark and evaluation methodology for LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• LLMs
• Qwen3.5-122B

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →