New VICIS task highlights VLM struggles with visual concept inference

Researchers have introduced VICIS, a new task designed to evaluate the ability of vision-language models (VLMs) to infer and apply visual concepts from sets of example images. Current state-of-the-art VLMs perform poorly on this task, often failing to utilize the visual context effectively or producing biased outputs. To address this, a novel training framework and architecture have been proposed that learn to extract concept-specific embeddings from image sets and queries, demonstrating improved accuracy and diversity in generating outputs, and generalizing to unseen concepts and modalities like sketches. AI

IMPACT This research highlights a current limitation in VLMs, potentially driving development towards models that can better understand and reason from visual context.

RANK_REASON The cluster contains an academic paper detailing a new task and proposed model for evaluating visual concept inference in VLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• arXiv
• Hugging Face
• ImageNet
• Nick Oliver Stracke
• vision-language models
• WordNet

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →