LLMs measure human values in social media texts via calibrated annotation

Researchers have developed a method to measure human values expressed in social media texts using LLMs. They utilized Schwartz's theory of basic human values and tested various LLMs and prompts on non-English social media posts. The study found that LLM interpretations of values can differ, but iterative prompt calibration and targeted expert verification rules improved accuracy and alignment with expert annotations. The LLM-annotated data was then successfully transferred to an encoder model for scalable prediction, retaining nuanced information about value expression and uncertainty. AI

IMPACT Provides a framework for understanding and quantifying subjective human values in large-scale text data, potentially aiding social science research and AI safety alignment.

RANK_REASON This is a research paper detailing a novel methodology for annotating and predicting human values from text using LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• LLM
• Schwartz's theory of basic human values
• Social Media

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →