A Controlled Audit of Pretraining Contamination in Public Medical Vision-Language Benchmarks
Researchers have audited public medical vision-language benchmarks for pretraining contamination, finding measurable image-side overlap on the SLAKE-En benchmark with models like SigLIP-B-16. Text analysis revealed canonical-order exchangeability signals in Qwen2.5-VL on SLAKE-En and other VLMs on OmniMedVQA. However, the study concluded that certain detection methods, like cohort-relative tail enrichment, are unreliable for small medical VLM cohorts. AI
IMPACT Highlights potential flaws in current VLM evaluation methods, necessitating more robust auditing for reliable medical AI development.