New CompoSET benchmark reveals vision-language models struggle with compositionality

Researchers have introduced CompoSET, a new benchmark designed to rigorously test vision-language compositionality in AI models. Unlike previous benchmarks that often change significant portions of an image, CompoSET creates 'foil' images that differ from a base image by only a single, controlled edit, such as a change in color, count, or spatial relation. This controlled approach aims to isolate the model's ability to understand compositional elements rather than general scene differences. Initial tests with 22 dual-encoder vision-language models revealed that all performed at or below chance level on CompoSET, indicating that current models may not possess the sophisticated compositional understanding previously assumed. AI

IMPACT Highlights a critical gap in current vision-language models' ability to understand nuanced compositional elements, potentially redirecting future research efforts.

RANK_REASON The cluster contains a new academic paper introducing a novel benchmark for AI research. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• alphaXiv
• arXiv
• CatalyzeX
• DagsHub
• Gotit.pub
• Hugging Face
• ScienceCast

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →