PulseAugur
EN
LIVE 06:46:03

New dataset VariNLI-VE reveals human label variation in visual entailment tasks

Researchers have introduced VariNLI-VE, a new dataset designed to study human label variation in Visual Entailment (VE) tasks. By re-annotating paired Natural Language Inference (NLI) and VE instances with the same annotators, they found that disagreement between human labels does not consistently decrease in VE compared to NLI, with some instances showing increased disagreement. The study also revealed that explanations for labels can vary even when the label itself remains the same. When evaluating vision-language models on this paired data, the models exhibited lower accuracy and less alignment with human label distributions in VE than in NLI. AI

IMPACT Highlights challenges in evaluating vision-language models and suggests areas for improvement in dataset creation and model alignment.

RANK_REASON The cluster describes a new dataset and research paper analyzing human label variation in visual entailment tasks. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →

New dataset VariNLI-VE reveals human label variation in visual entailment tasks

How we ranked this

Signal score
40 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Tool
The cluster describes a new dataset and research paper analyzing human label variation in visual entailment tasks. [lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration
Single-source cluster
Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics
paper, other
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
Breaking (< 6h)
Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

Full methodology in our editorial standards.

COVERAGE [1]

  1. arXiv cs.CL TIER_1 English(EN) · Monica Riedler, Jasmin Orth, Raffaella Bernardi, Barbara Plank ·

    In the Eye of the Beholder: Human Label Variation in Textual and Visual Entailment

    arXiv:2611.17954v1 Announce Type: new Abstract: Human Label Variation (HLV) is increasingly recognised as an informative signal that reflects meaningful differences in how people interpret the same input. While HLV has been studied extensively in Natural Language Inference (NLI),…