LLM measurements lack construct validity despite high reproducibility, study finds

A new research paper published on arXiv explores the distinction between reproducibility and construct validity in Large Language Model (LLM) measurements. The study used data from the European Commission's AI Act consultation, finding that while LLM annotations of text submissions were highly reproducible, they did not consistently align with survey-reported measures of the same constructs. This divergence varied across stakeholder groups, with business associations showing a greater concern for AI risks in their written submissions compared to survey responses. The research highlights the importance of validating LLM measurements beyond mere reproducibility, considering construct validity and communication context. AI

影响 Highlights potential pitfalls in using LLMs for research measurement, emphasizing the need for careful validation beyond simple reproducibility.

排序理由 Academic paper on LLM measurement methodology. [lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.AI 阅读 →

• AI Act
• Carlo Romano Marcello Alessandro Santagiustina
• European Commission
• Large Language Model

• 论文
• 安全

AI 生成摘要 · Google Gemini · 来自 1 个来源。 我们如何撰写摘要 →