LLMs struggle to replicate novel human insights in synthetic survey data

A new arXiv paper compares five leading LLMs—ChatGPT Thinking 5 Pro, Claude Sonnet 4.5 Pro, Claude CoWork 1.123, Gemini Advanced 2.5 Pro, Incredible 1.0, and DeepSeek 3.2—in their ability to replicate human survey responses using synthetic data. The study found that while these models can generate plausible and harmonized results, they fail to capture novel or counterintuitive insights present in human data. The research suggests that current LLMs are adept at echoing conventional wisdom but not at uncovering unique findings, highlighting the need for robust validation protocols for responsible use of synthetic survey data. AI

IMPACT Highlights limitations of LLMs in generating novel insights, suggesting synthetic data should supplement, not replace, human research.

RANK_REASON The cluster contains an academic paper detailing research findings on LLM capabilities. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• ChatGPT Thinking 5 Pro
• Claude CoWork 1.123
• Claude Sonnet 4.5 Pro
• DeepSeek 3.2
• Gemini Advanced 2.5 Pro
• Incredible 1.0
• Silicon Valley

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →