New dataset probes VLM concept learning against human generalization

Researchers have introduced the Novel Visual References Dataset (NVRD), comprising over 19,000 images across 90 visual concepts, designed to test how vision-language models (VLMs) learn new concepts, especially when they conflict with pre-existing knowledge. Evaluations of both open- and closed-source models alongside human judgments revealed that VLMs struggle to adapt to novel concepts in-context and tend to overgeneralize learned labels to incorrect stimuli, unlike humans. The NVRD aims to serve as a benchmark for studying visual concept acquisition in both humans and machines. AI

IMPACT Establishes a new benchmark for evaluating VLM concept learning and generalization, highlighting current limitations compared to human capabilities.

RANK_REASON The cluster contains an academic paper detailing a new dataset and benchmark for evaluating vision-language models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• Novel Visual References Dataset
• NVRD
• vision-language models

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →