CIR benchmarks overestimate AI model composition abilities

A new research paper questions the effectiveness of current benchmarks for Composed Image Retrieval (CIR), a task requiring models to combine image and text information. The study found that many existing CIR benchmarks can be solved using only one modality, indicating models are exploiting "unimodal shortcuts" rather than truly composing information. After auditing and validating queries, researchers re-evaluated models on a cleaner subset, revealing a greater reliance on multimodal composition but also a decrease in accuracy, suggesting current benchmarks overestimate model capabilities. AI

Summary written by gemini-2.5-flash-lite from 1 source. How we write summaries →

IMPACT Reveals limitations in current AI evaluation methods for multimodal tasks, potentially guiding future benchmark development.

RANK_REASON Academic paper analyzing benchmark limitations for a specific AI task. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• Composed Image Retrieval
• Generalist Multimodal Embedding models
• Matteo Attimonelli

• paper
• other