A new benchmark study has revealed that current open-weight large language models are unreliable for recalling specific religious texts, such as Hadith. The research tested five models, including Mistral-7B and Gemma3-4B, across six canonical Sunni Hadith collections, generating 30,000 responses in both English and Arabic. Astonishingly, none of the responses achieved both high lexical and semantic fidelity, meaning none provided an exact, verifiable match to the source text. Arabic responses performed worse than English, suggesting that LLMs without retrieval augmentation are potentially dangerous for Islamic scholarship and should not be relied upon for parametric memory. AI
IMPACT Current open-weight LLMs are unreliable for exact religious text recall, necessitating retrieval-augmented generation for such high-stakes applications.
RANK_REASON The cluster contains an academic paper detailing a new benchmark for evaluating LLM performance on a specific task. [lever_c_demoted from research: ic=1 ai=1.0]
- ALLaM
- Gemma3-4B
- Jais-7B
- Jami' al-Tirmidhi
- Mistral-7B
- Muhammad
- Phi4-mini-3.8B
- Sahih al-Bukhari
- Sahih Muslim
- Sunan Abi Da'ud
- Sunan ibn Majah
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
