A new study analyzing the Humanity's Last Exam (HLE) benchmark has found that its multiple-choice subset, comprising 428 items, primarily measures a single general reasoning factor rather than distinct subject-domain capabilities. Researchers applied psychometric methods, including an IRT model, to evaluate 29 large language models. The findings indicate that domain labels explain a minimal amount of variance, and domain-specific ability estimates are highly redundant with the total score. Furthermore, the benchmark's measurement precision is concentrated at moderate ability levels, limiting its ability to effectively differentiate among the most advanced frontier models. AI
IMPACT This research suggests current benchmarks may not accurately assess the distinct capabilities of advanced LLMs, potentially impacting future model development and evaluation strategies.
RANK_REASON Academic paper analyzing an existing LLM benchmark. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
