How Many Human Survey Respondents is a Large Language Model Worth? An Uncertainty Quantification Perspective
Researchers have developed a new framework to quantify the uncertainty in using large language models (LLMs) to simulate survey responses. The method helps determine the optimal number of simulated responses needed to ensure reliable inference about population parameters, balancing the risk of overly narrow or overly wide confidence sets. This approach adaptively selects the simulation sample size to achieve nominal coverage, regardless of the LLM's accuracy, and can also reflect the LLM's simulation fidelity. AI
IMPACT Provides a method to improve the reliability of survey data generated by LLMs, potentially impacting market research and social science.