A new benchmark called ChronoCulture has been developed to evaluate the cultural awareness of large language models (LLMs) across different time periods. The benchmark consists of 10,360 temporal pairs derived from longitudinal survey data across 72 countries, using survey questions and population response distributions as ground truth. Initial testing on six LLMs revealed that while these models can often predict cultural attitudes at a specific point in time, they struggle to accurately represent how those attitudes evolve. Prompting the models with a specific year also showed minimal improvement in aligning with survey responses, indicating a general tendency for LLMs to present a static view of cultural dynamics. AI
IMPACT Highlights a gap in LLM understanding of temporal cultural shifts, suggesting future models need to better capture evolving societal norms.
RANK_REASON The cluster describes a new academic paper introducing a novel benchmark for evaluating LLMs. [lever_c_demoted from research: ic=1 ai=1.0]
- arXiv
- ChronoCulture
- country
- Hugging Face
- large language models
- longitudinal survey data
- population response distributions
- six LLMs
- survey question
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
