Researchers have developed a new benchmark called Trick-or-Treat to evaluate multimodal medical reasoning in frontier AI models, specifically focusing on their ability to interpret paired chest radiographs and clinical photographs. This benchmark utilizes synthetic clinical photographs to create adversarial scenarios where the photograph's findings are intentionally mismatched with the radiograph's, revealing significant hallucination behaviors in current models. When presented with these mismatched pairs, frontier models incorrectly identified clinically significant findings in the photographs between 45% and 57.5% of the time, often overriding the radiograph's information to align with a hallucinated photographic diagnosis. AI
IMPACT Highlights potential risks in AI's medical reasoning capabilities, suggesting a need for more robust adversarial testing.
RANK_REASON The cluster contains a research paper detailing a new benchmark for evaluating AI models. [lever_c_demoted from research: ic=1 ai=1.0]
- acne
- arXiv
- chickenpox
- Clinical photographs: the gold standard, an update
- Radiographs Illustrating the Report of the Shoe Board Meeting at Fort Leavenworth, Kansas (NAID 4700545)
- Trick or Treat
- Varicella pneumonia
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
