PulseAugur
EN
LIVE 06:26:09

New framework tests LLMs' ability to simulate survey respondents

By PulseAugur Editorial · [2 sources] ·

Researchers have developed a new evaluation framework called cross-survey transfer to assess the effectiveness of large language models (LLMs) in simulating human survey respondents. Using data from the Taiwan Election and Democratization Study (TEDS) 2024, the study found that LLMs with 27B-120B parameters achieved 52% accuracy in predicting responses to unseen survey questions, performing comparably to supervised machine learning models. The research also identified a hierarchy of predictability across different types of survey items, with attitudes being more predictable than sovereignty. Furthermore, the study nuanced previous findings on limitations like variance collapse and safety alignment, suggesting these issues are not exclusive to LLMs. AI

IMPACT This research clarifies the capabilities and limitations of using LLMs for survey simulation, potentially impacting how social science research is conducted.

RANK_REASON Academic paper published on arXiv detailing a new evaluation framework for LLMs in survey research. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.MA (Multiagent) →

AI-generated summary · Google Gemini · from 2 sources. How we write summaries →

New framework tests LLMs' ability to simulate survey respondents

COVERAGE [2]

  1. arXiv cs.AI TIER_1 English(EN) · Chan-Tung Ku, Chan Hsu, Pei-Cing Huang, Frank Cheng-shan Liu, I-Ling Cheng, Yihuang Kang ·

    Silicon Sampling via Cross-Survey Transfer

    arXiv:2607.03091v1 Announce Type: new Abstract: Silicon sampling-using large language models (LLMs) to simulate human survey respondents-has emerged as a promising approach for augmenting traditional survey research. However, most evaluations rely on distributional comparisons ra…

  2. arXiv cs.MA (Multiagent) TIER_1 English(EN) · Yihuang Kang ·

    Silicon Sampling via Cross-Survey Transfer

    Silicon sampling-using large language models (LLMs) to simulate human survey respondents-has emerged as a promising approach for augmenting traditional survey research. However, most evaluations rely on distributional comparisons rather than individual-level prediction, which ris…

RELATED ENTITIES

RELATED TOPICS