LLM annotation sensitivity to concept definitions highlighted in new study

A new study published on arXiv investigates how Large Language Models (LLMs) interpret and apply definitions when annotating social science concepts. Researchers found that LLMs exhibit "definition sensitivity," meaning their annotations change significantly when presented with different, competing definitions of the same concept. This sensitivity was more pronounced than when definitions were merely paraphrased, indicating that LLM-generated annotations are heavily influenced by the specific wording of the provided definition. The findings suggest that researchers using LLMs for annotation must report the exact definitions used, as the LLM's interpretation can alter the resulting labels. AI

IMPACT Highlights the need for careful prompt engineering and transparent reporting when using LLMs for research annotation.

RANK_REASON The cluster contains a research paper published on arXiv detailing experimental findings about LLM behavior. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• alphaXiv
• arXiv
• CatalyzeX Code Finder for Papers
• Connected Papers
• CORE Recommender
• DagsHub
• Gotit.pub
• Hugging Face
• Influence Flower
• Large Language Models
• Litmaps
• LLMs
• ScienceCast
• scite Smart Citations

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →