New IRT model predicts LLM abilities efficiently with 85% cost reduction

Researchers have developed a cost-efficient method for evaluating large language models (LLMs) by predicting their performance on unseen tasks. This approach utilizes a modified multidimensional item response theory (IRT) model combined with adaptive item selection. The method, tested on a dataset of 65 models and over 100,000 items, can predict performance on held-out benchmarks with a mean absolute error of less than 7% after observing only 16 items. Furthermore, by incorporating cost-aware factors, the evaluation cost can be reduced by up to 85%. AI

IMPACT This research offers a more efficient and cost-effective way to benchmark LLMs, potentially accelerating model development and comparison.

RANK_REASON The cluster contains an academic paper detailing a new methodology for evaluating LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• Irt
• Michael Krumdick
• Wide-scale Item Level Dataset
• Wild

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →