New NLU guideline reveals high label error rates, improved by LLMs

Researchers have identified significant label errors in datasets for individual-level Natural Language Understanding (NLU) tasks like stance detection and sentiment analysis, with error rates reaching up to 31.7%. To address this, they propose a new annotation guideline that incorporates an individual's other posts to better capture subjective perspectives. Experiments using this guideline and large language models like GPT-4o and Llama3-70B showed that incorporating individual factors improved model accuracy to over 87% on re-annotated datasets, highlighting the importance of this approach for future NLU dataset creation. AI

IMPACT Highlights the need for improved data annotation for LLMs in subjective NLU tasks, potentially improving model performance on nuanced human language.

RANK_REASON Research paper detailing a new methodology and experimental results. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• GPT-4o
• Llama3 70B
• Nature Language Understanding
• sentiment analysis
• stance detection
• Yunpeng Xiao

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →