NLP research often omits key human annotation details, study finds

A new study published on arXiv assesses the reporting of human annotation practices in Natural Language Processing (NLP) research from 2018 to 2025. The research found that while operational details like recruitment and annotation volume are frequently documented, crucial information for assessing validity, such as annotator training, compensation, and agreement metrics, is often omitted. The study introduces a framework and recommendations to improve the reliability and interpretability of human annotation in NLP. AI

IMPACT Highlights critical gaps in reporting for NLP research, potentially impacting the reliability and reproducibility of AI models trained on human-annotated data.

RANK_REASON Academic paper analyzing reporting practices in a research field. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• NLP

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →