New benchmark GesBench evaluates incremental social perception in multimodal LLMs

Researchers have introduced GesBench, a new benchmark designed to evaluate how well multimodal large language models (LLMs) understand social meaning incrementally, similar to how humans process information. Unlike existing benchmarks that focus on final answers, GesBench uses cumulative clips of utterances to assess when models form interpretations of speaker intent and social affordance. The benchmark also tests whether models attend to the same cues humans do by corrupting regions guided by human gaze. Initial findings reveal a significant gap between human and model perception, with humans forming predictive understandings earlier and revising them more frequently, while models tend to be more hesitant. AI

IMPACT This benchmark could drive the development of LLMs with more nuanced social understanding, crucial for human-AI interaction.

RANK_REASON The item describes a new academic paper introducing a novel benchmark for evaluating multimodal language models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• alphaXiv
• arXiv
• CatalyzeX
• DagsHub
• GesBench
• Gotit.pub
• Hugging Face
• Influence Flower
• multimodal language models
• ScienceCast

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →