VisAnalog: A Diagnostic Suite for Visual Concept Transfer on Natural Images
Researchers have introduced VisAnalog, a new diagnostic suite designed to evaluate how well visual models can transfer concepts across different images and transformations. The benchmark consists of 617 human-validated questions that test a model's ability to recognize and manipulate visual properties through steps like rotation, flipping, and color changes. Initial tests on various vision-language models revealed significantly lower accuracy compared to human performance, particularly as the complexity of transformations increased, indicating a primary bottleneck in relation inference. AI
IMPACT Introduces a new benchmark to identify weaknesses in visual concept transfer, potentially guiding future model development.