New IslamicMMLU benchmark evaluates LLMs on Quran, Hadith, and Fiqh

Researchers have developed IslamicMMLU, a new benchmark designed to evaluate the performance of large language models on core Islamic knowledge across three disciplines: Quran, Hadith, and Fiqh. The benchmark comprises over 10,000 multiple-choice questions and was used to assess 26 LLMs, with accuracy scores ranging from 39.8% to 93.8%. Notably, Gemini 3 Flash achieved the highest average accuracy, while Arabic-specific models showed mixed results and generally underperformed compared to frontier models. The benchmark also includes a task to detect bias related to different Islamic jurisprudence schools. AI

IMPACT This benchmark will enable more precise evaluation of LLMs' capabilities in understanding and responding to Islamic queries, potentially leading to improved models for religious and cultural contexts.

RANK_REASON The cluster describes a new academic benchmark for evaluating LLMs on a specific domain of knowledge. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• Ali S Abdelaal
• Arabic-specific models
• arXiv
• fiqh
• Gemini 3 Flash
• hadith
• Hugging Face
• IslamicMMLU
• Qur’an

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →