New testbed Bench-C reveals reliability flaws in vision-language models

Researchers have developed Bench-C, a new testbed designed to evaluate the reliability of vision-language models (VLMs) when exposed to visual corruptions. This testbed goes beyond simple accuracy metrics to analyze how corruptions affect a model's internal prediction structure and confidence. Bench-C utilizes 19 corruption types across five severity levels, introducing a Robustness Alignment Score (RAS) to measure the alignment between confidence and correctness. Experiments with 13 VLMs revealed that even mild corruptions can sometimes improve top-1 accuracy while simultaneously degrading prediction stability and increasing overconfidence. AI

IMPACT Highlights the need for more robust evaluation methods for AI models beyond simple accuracy metrics.

RANK_REASON Research paper introducing a new benchmark/testbed for evaluating AI models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• Vision-Language Models
• Xiangjie Sui

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →