Researchers have introduced XYBench, a new benchmark designed to evaluate Large Language Models' (LLMs) ability to handle queries containing misconceptions, a common issue known as the XY-problem. The benchmark comprises 8,115 queries from both technical and everyday domains, assessing LLMs on their identification of misconceptions and their provision of pragmatic, intended solutions. Experiments reveal that even leading LLMs frequently address the literal request rather than the underlying problem and struggle significantly with identifying user misconceptions compared to humans. AI
IMPACT Highlights a key limitation in current LLMs, suggesting a need for improved pragmatic reasoning and user intent understanding.
RANK_REASON The item describes a new academic paper introducing a benchmark for evaluating LLM capabilities. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
