New benchmark reveals LLMs struggle to simulate realistic customer behavior

Researchers have introduced CustomerSim, a new benchmark designed to evaluate how well multimodal large language models (MLLMs) can simulate realistic customer behavior in chat-based retail scenarios. The benchmark includes 360 curated personas across five product categories and metrics for assessing consistency and conversational quality. Initial tests revealed that current state-of-the-art models, including Claude Opus-4.8 and GPT-5.6 "Sol", struggle with persona adherence and exhibit lower lexical diversity than human shoppers. To address these limitations, the team developed UserGRPO, a reinforcement learning approach that significantly improves decision alignment without sacrificing conversational quality. AI

IMPACT This benchmark could accelerate the development of more sophisticated AI agents capable of nuanced, goal-oriented interactions in commercial settings.

RANK_REASON The cluster describes a new academic paper introducing a benchmark and a novel method for evaluating LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• Claude Opus-4.8
• CustomerSim
• GPT-5.6 "Sol"
• MLLMs
• Multimodal Large Language Models and Tunings: Vision, Language, Sensors, Audio, and Beyond
• UserGRPO
• Yada Pruksachatkun

• paper
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →