New EYT-Bench evaluates LLM dialogue, revealing intent tracking gaps

A new benchmark called EYT-Bench has been developed to evaluate large language models (LLMs) on multi-turn dialogue capabilities, focusing on persona consistency, intent tracking, and goal completion. The benchmark utilizes a decoupled design with a user simulator, a target model, and an LLM judge, drawing personas from human-curated corpora to minimize bias. Initial evaluations reveal that while current models perform similarly on subjective conversational aspects, they vary significantly in objective intent tracking. The research also highlights that reasoning abilities improve objective tracking, persona format impacts performance, and most models exhibit a warm-up effect, with GPT-5.5 being a notable exception. AI

IMPACT This benchmark could drive improvements in LLM dialogue systems by highlighting weaknesses in intent tracking and persona consistency.

RANK_REASON The cluster contains a research paper introducing a new benchmark for evaluating LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• DeepSeek V4-Pro
• EYT-Bench
• Gemma 4
• GPT-5.5
• Nemotron-Personas-USA
• PersonaMem-v2
• tau-bench

• paper
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →