Researchers have introduced FriendBench, a new benchmark designed to evaluate the ability of both humans and multimodal large language models (LLMs) to infer familiarity between two people based on a short conversation clip. The benchmark uses 20-second video, audio, or text segments of dyadic ice-breaker conversations. Across various modalities, 26 models from seven companies were compared against human panels, with the top-performing models achieving accuracy indistinguishable from humans. However, a key difference emerged: while humans maintained a balanced prediction, the strongest models showed a tendency to predict "stranger," suggesting a difference in their effective prior assumptions rather than discriminatory ability. AI
IMPACT This benchmark could advance research into social intelligence in AI, potentially leading to more nuanced and context-aware multimodal models.
RANK_REASON The cluster describes a new academic paper introducing a benchmark for evaluating LLMs and humans on a specific task. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
