LLMs match trained linguists on complex annotation tasks, study finds

A new paper published on arXiv explores the challenges of linguistic annotation, comparing human annotators with large language models (LLMs). Researchers analyzed evaluative language in TED talk transcripts, focusing on the Appraisal theory's subcategories of Affect, Judgement, and Appreciation. The study found that LLMs, when prompted effectively, performed comparably to a trained linguist, achieving an F1-score of 0.77, and outperformed linguists in training. AI

IMPACT LLMs show promise in aiding complex annotation tasks, potentially accelerating research in digital humanities and linguistics.

RANK_REASON The cluster contains an academic paper detailing a research study on LLM capabilities. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• Aenne Cecilia Knierim
• affect
• Appraisal theory
• arXiv
• F1 score
• gratitude
• Hugging Face
• Judgement
• large language models
• TED talk

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →