LLM persuasion evaluations show weak agreement across methods

A new study published on arXiv evaluates the persuasive capabilities of fifteen large language models (LLMs), finding that different evaluation methods yield weakly correlated results. The study adapted nine existing automated methods to a shared setup and discovered that model refusals, particularly on manipulation tasks, significantly reduce agreement between these methods. General capability also plays a role, with rational persuasion methods tracking it while manipulation methods do not, suggesting that a single persuasion score is task-specific and does not reflect a model's overall persuasiveness. AI

IMPACT Highlights the challenge in reliably evaluating LLM persuasion, impacting safety and alignment research.

RANK_REASON The cluster contains a research paper published on arXiv detailing an evaluation of LLM capabilities. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• alphaXiv
• arXiv
• CatalyzeX
• Connected Papers
• Core Recommendations for Antifungal Stewardship: A Statement of the Mycoses Study Group Education and Research Consortium
• DagsHub
• Gotit.pub
• Hugging Face
• Kamile Dementaviciute
• Litmaps
• ScienceCast
• scite Smart Citations

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →