New benchmark reveals bias in multimodal LLM judges for image editing

Researchers have developed a new benchmark called EditJudgeBias to audit biases in multimodal large language models (MLLMs) when they are used as judges for image editing tasks. The benchmark includes 1,196 real editing samples and 13 injected cues designed to test for quality-preserving interventions. Experiments revealed that MLLM judges are susceptible to irrelevant visual cues, with fabricated majority opinions increasing ratings and changes in candidate order reversing a significant percentage of decisions. The study highlights that different metrics are needed to characterize MLLM judge robustness, as a single metric cannot capture all aspects of their performance. AI

IMPACT Highlights potential biases in LLM-based evaluation systems, crucial for reliable AI training and assessment.

RANK_REASON The cluster contains an academic paper detailing a new benchmark and experimental findings. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• arXiv
• EditJudgeBias
• Hugging Face
• multimodal large language model

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →