New benchmark PerSo tests LLM social intelligence in real-world persuasion

Researchers have introduced PerSo, a new benchmark designed to evaluate the social intelligence of large language models (LLMs) in real-world persuasion scenarios. The benchmark assesses both cognitive intelligence, the ability to infer mental states, and behavioral intelligence, the ability to choose appropriate social actions. PerSo utilizes dialogues from Reddit's ChangeMyView community and an LLM-in-the-loop pipeline for task construction, validated by human annotation. Evaluations show that while larger models generally perform better, the best LLM slightly exceeds human performance on behavioral tasks, but humans still outperform LLMs on cognitive inference. AI

IMPACT This benchmark could drive development of more socially aware and effective LLMs for human-AI interaction.

RANK_REASON The item describes a new academic paper introducing a benchmark for evaluating LLM capabilities. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• alphaXiv
• arXiv
• CatalyzeX
• ChangeMyView
• DagsHub
• Gotit.pub
• Hugging Face
• Influence Flower
• LLMs
• Reddit
• ScienceCast

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →