Formal semantic structure explains minimal human label variation in NLI tasks

A new research paper explores the extent to which formal semantic structure explains human label variation in natural language inference (NLI) tasks. The study analyzed items from the SNLI and MNLI corpora, finding that hypotheses with less straightforward monotonicity exhibit higher label entropy. However, the formal profiles accounted for only a small percentage of entropy variance, indicating they are insufficient for identifying items with high annotator disagreement. The research also found that semantic structure did not significantly alter the nature of disagreements among annotators. AI

IMPACT This research suggests current formal semantic structures are insufficient for fully understanding or predicting human disagreement in NLI tasks, potentially impacting the development of more robust and interpretable models.

RANK_REASON Academic paper detailing a new analysis of existing NLI datasets. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• ChaosNLI
• Hugging Face
• MNLI
• Stanford Natural Language Inference corpus
• VariErr NLI: Separating annotation error from human label variation

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →