New BRUCE framework benchmarks VLM robustness under image corruption

Researchers have introduced BRUCE, a new framework designed to evaluate the robustness of visual-language models (VLMs) when faced with corrupted or distorted input images. Unlike existing methods that primarily focus on clean-task accuracy, BRUCE quantifies how rapidly a VLM's reasoning performance degrades as visual corruption severity increases. The framework utilizes novel metrics, the Robustness Corruption Index (RCI) and Traversal-RCI (T-RCI), to measure this deterioration across various scientific reasoning tasks, including OCR-dependent, spatial, and symbolic reasoning. AI

IMPACT This framework could lead to more reliable visual-language models by highlighting their failure modes under realistic, degraded input conditions.

RANK_REASON The cluster describes a new academic paper introducing a novel benchmarking framework for evaluating AI model robustness. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• arXiv
• BRUCE
• DagsHub
• Hugging Face
• Robustness Corruption Index
• T-RCI

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →