Researchers have developed MIRA, a new bilingual benchmark designed to evaluate how well large language models (LLMs) maintain consistent medical information across different phrasings of the same question. The benchmark, comprising 4,320 prompts derived from 60 health questions, revealed that LLMs often provide less comprehensive information and fewer actionable steps when prompts are phrased with lower health literacy. This phenomenon, termed Differential Information Dilution (DID), was observed to be model-specific, with some models like Claude and Qwen showing improvements when prompted with knowledge-guided mitigation techniques. AI
IMPACT Highlights potential risks in LLM-driven health information, prompting developers to improve consistency and reduce information dilution.
RANK_REASON This is a research paper introducing a new benchmark for evaluating LLMs. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
