LLMs over-credit same-trial evidence in medical verification tasks

A new arXiv paper reveals that large language models, including GPT-5.5, Qwen3-4B, and Ministral 3 8B, struggle with evidence aggregation in medical verification tasks. While these models can correctly identify shared trial lineage, they tend to over-credit publications from the same trial when synthesizing information. This behavior was observed across multiple provenance families, though a specific instruction to group studies reduced the over-crediting in GPT-5.5 for certain families. The findings highlight a discrepancy between a model's ability to recognize study lineage and its capacity to aggregate evidence effectively. AI

IMPACT Highlights a critical gap in LLM reasoning for complex tasks like medical verification, potentially impacting their reliability in high-stakes applications.

RANK_REASON Research paper detailing limitations of LLMs in evidence aggregation. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• alphaXiv
• arXiv
• CatalyzeX
• Connected Papers
• CORE Recommender
• DagsHub
• Gotit.pub
• GPT-5.5
• Hugging Face
• Influence Flower
• Litmaps
• Ministral 3 8B
• Qwen3-4B
• ScienceCast
• scite Smart Citations

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →