New ABACUS benchmark reveals image editing models struggle with object counting

Researchers have introduced ABACUS, a new benchmark and evaluation protocol designed to accurately assess the numerical capabilities of image editing models. Current methods often overlook errors in object counting and scene preservation, especially in complex scenes or with higher requested counts. ABACUS employs a judge-free protocol that verifies edits at an instance level against ground truth, ensuring the correct number of objects is present and the scene remains intact, thereby revealing significant numerical failures in leading image editors. AI

IMPACT This benchmark will enable more accurate evaluation of image editing models, driving improvements in their ability to precisely manipulate object counts and maintain scene consistency.

RANK_REASON The cluster describes a new academic benchmark and evaluation protocol for AI models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• ABACUS
• arXiv
• multimodal large language model
• vision-language model

• paper
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →