New attacks reveal vision-language model confidence is not robust

Researchers have developed new methods to attack the confidence scores of vision-language models, demonstrating that these scores are not inherently robust. The study found that various attacks, including image-only perturbations and hidden-state interventions, can manipulate confidence readouts without altering the generated answer. These attacks significantly reduce the accuracy of confidence-gated systems, sometimes even below a baseline without confidence gating. The findings suggest that current confidence mechanisms in deployed systems are sensitive to integrity issues rather than intrinsically robust. AI

IMPACT Demonstrates that current confidence mechanisms in vision-language models are vulnerable, potentially impacting their reliability in safety-critical applications.

RANK_REASON Academic paper detailing novel attack methods on AI model confidence. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• answer-string accuracy
• confidence channels
• correctness-label-aware attacks
• defense estimators
• defense families
• hidden-state interventions
• text-model activation-space replication
• token-probability attack
• uniform amplitude certificate
• Vision-Language Systems
• visual question answering benchmarks

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →