New benchmark tests AI's social reasoning in counterfactual videos

Researchers have introduced SocialReasonBench, a new video question-answering benchmark designed to evaluate the social reasoning capabilities of Large Multimodal Models (LMMs). This benchmark utilizes counterfactual narrative videos from the game Detroit: Become Human, where player choices lead to different social outcomes. SocialReasonBench assesses seven reasoning dimensions, including intent recognition, emotional empathy, and counterfactual reasoning, revealing that current LMMs perform well on basic social understanding but struggle with more complex causal and counterfactual scenarios. AI

IMPACT Highlights limitations in current AI models' ability to understand complex social dynamics and counterfactual scenarios, driving future research directions.

RANK_REASON The cluster contains a research paper introducing a new benchmark for AI evaluation. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• Detroit: Become Human
• Hugging Face
• Large Multimodal Models
• SocialReasonBench

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →