New PersianMedQA benchmark tests LLMs on bilingual medical reasoning

Researchers have introduced PersianMedQA, a new benchmark dataset designed to evaluate the medical question-answering capabilities of large language models (LLMs) in both Persian and English. The dataset comprises over 20,000 expert-validated multiple-choice questions from Iranian medical exams, covering 23 specialties. Benchmarking 41 models, the study found that closed-weight general models like GPT-4.1 performed best, while specialized Persian LLMs struggled. The research also highlighted that some medical nuances are lost in translation, making Persian-specific answers crucial. AI

IMPACT This benchmark will drive improvements in LLM performance for low-resource languages and specialized domains like medicine.

RANK_REASON The cluster describes a new academic paper introducing a benchmark dataset for evaluating LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• Dorna
• GPT-4.1
• Large Language Models
• Mohammad Javad Ranjbar Kalahroodi
• PersianMedQA

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →