A new research paper introduces a method to evaluate the true capabilities of large language models on multiple-choice legal benchmarks, addressing the issue of models exploiting question-agnostic answer patterns. The study found that models like Claude Haiku 4.5 and GPT-5.6 exhibit significant biases towards certain answer positions, even when the question is hidden. By implementing a 'gating' mechanism that filters out items with such biases, the researchers revealed that the true performance of many models is closer to chance, highlighting the importance of benchmark design in accurately assessing model capabilities. AI
IMPACT Highlights flaws in current LLM evaluation methods, potentially leading to more robust benchmark design and a clearer understanding of model capabilities.
RANK_REASON Research paper introducing a new methodology and benchmark analysis for LLMs. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
