Vision-language models struggle with institutional agreement on chest X-rays

Researchers have evaluated three vision-language models (VLMs) on their ability to accurately identify chest radiograph findings across different institutions. The study found that existing VLMs lack confidence scores, making it difficult for receiving institutions to gauge the trustworthiness of individual predictions. When tested on over 345,000 predictions across three corpora and six findings, even with a small budget of local labels for estimation, the models' performance varied significantly by site and interface, indicating a need for site-specific re-evaluation rather than a universal default estimator. AI

IMPACT Highlights the need for site-specific calibration and confidence scoring in medical AI to ensure reliable deployment across different healthcare settings.

RANK_REASON The cluster contains an academic paper detailing research findings on the performance of AI models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• Beta-Binomial empirical-Bayes estimator
• CORE Recommender
• Hugging Face
• logistic regression model
• vision-language model

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →