From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation
Researchers have developed a new method called Conformal Elo Estimation to improve the evaluation of large language models (LLMs). This technique addresses systematic errors in LLM-as-a-judge evaluations, such as position bias and self-preference, by propagating calibrated win probabilities into the Elo estimation process. The method significantly reduces the mean absolute error between LLM-derived and human-derived ratings, bringing them within 17.9 Elo MAE. Additionally, it applies conformal prediction to provide honest uncertainty bounds, offering a low-cost tool for developers to obtain calibrated LLM estimates without extensive human annotation. AI
IMPACT Provides a more accurate and cost-effective way to evaluate LLMs, enabling better model development and comparison.