A new research paper highlights significant unreliability issues with using large language models (LLMs) as judges for evaluating AI outputs. The study found that the same request sent to the same model endpoint can produce different rankings over time, with same-window repeat rankings agreeing at a Spearman correlation of only 0.400 and next-day replays at 0.78, far below the required 0.90 and 0.99 respectively. The research identified three primary causes: a biased label-to-meaning mapping, candidate gaps far smaller than the instrument's noise floor, and inherent variability in model responses. These issues persisted even when switching providers or attempting to mitigate them through sampling, suggesting that model names on shared infrastructure do not represent stable measurement instruments. AI
IMPACT Highlights critical issues in evaluating LLM outputs, potentially impacting leaderboards, training data selection, and model development.
RANK_REASON Academic paper detailing a reliability failure in LLM measurement tools. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
