New framework integrates visual and language models for chest X-ray classification

Researchers have developed a new framework for multi-label chest X-ray classification that integrates unimodal visual representations from RAD-DINO with vision-language representations from BioViL-T. This approach refines and fuses these embeddings in latent space, aiming to improve classification performance and understand the complementary roles of each data source. Experiments on the MIMIC-CXR-JPG dataset demonstrated that while RAD-DINO outperformed BioViL-T independently, their combined use, particularly with hybrid fusion after individual refinement, yielded the best results, achieving a mean AUROC of 0.840 and an mAP of 0.467. The study acknowledges that further validation is needed to confirm generalizability to data from other institutions. AI

IMPACT This research could lead to more accurate and nuanced diagnostic tools in medical imaging by better leveraging diverse data modalities.

RANK_REASON The cluster contains an academic paper detailing a new methodology for a specific AI task. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• arXiv
• BioViL-T
• MIMIC-CXR-JPG
• RAD-DINO

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →