A new benchmark called InferBench has been developed to evaluate how well large language models infer user priorities from instructions. In tests involving 2.8k conversations across 12 LLMs, models that clearly understood stated preferences achieved 89% accuracy, while those with unstated preferences dropped to 60%. GPT-6 Astra performed best at 76%, closely followed by MiMo V2.6 Pro at 75%, and Gemini 3.1 Pro at 69%. The benchmark also highlighted that LLMs remain highly confident even when incorrect, with over 100 wrong decisions made with more than 90% confidence. AI
IMPACT Highlights the importance of instruction following and user intent understanding for LLMs, with implications for user experience and model development.
RANK_REASON New benchmark and evaluation results for LLMs. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
