Researchers have introduced VariNLI-VE, a new dataset designed to study human label variation in Visual Entailment (VE) tasks. By re-annotating paired Natural Language Inference (NLI) and VE instances with the same annotators, they found that disagreement between human labels does not consistently decrease in VE compared to NLI, with some instances showing increased disagreement. The study also revealed that explanations for labels can vary even when the label itself remains the same. When evaluating vision-language models on this paired data, the models exhibited lower accuracy and less alignment with human label distributions in VE than in NLI. AI
IMPACT Highlights challenges in evaluating vision-language models and suggests areas for improvement in dataset creation and model alignment.
RANK_REASON The cluster describes a new dataset and research paper analyzing human label variation in visual entailment tasks. [lever_c_demoted from research: ic=1 ai=1.0]
- arXiv
- Bowman et al.
- Hugging Face
- Human Label Variation
- Stanford Natural Language Inference corpus
- Natural Language Inference
- VariNLI-VE
- vision-language model
- Visual Entailment
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →