Researchers have introduced SocietyBench, a new benchmark designed to evaluate how well large language models can forecast the evolution of social events. The benchmark works by collecting news and social media posts, distilling them into a timeline of factual events and public opinion, and then generating forecasting questions. To prevent models from relying on pre-training memory, the benchmark anonymizes named entities and shifts dates, creating a counterfactual social world. Early results show that even the strongest frontier LLMs struggle, achieving only a score of 75.0 out of 100 on two axes: probability calibration and temporal accuracy. AI
IMPACT This benchmark could drive development of LLMs with better understanding and forecasting capabilities for real-world social dynamics.
RANK_REASON The item describes a new academic benchmark for evaluating LLMs. [lever_c_demoted from research: ic=1 ai=1.0]
Read on Hugging Face Daily Papers →
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
