A local large language model used as a judge for evaluating responses demonstrated significant but weak correlation with human ratings, failing to meet the 0.5 Spearman correlation threshold with a score of 0.395. While the model showed strong internal consistency, achieving a repeat-to-repeat variation score well below the 0.75 point threshold, it frequently diverged from human judgments. In one instance, a response rated as extremely helpful by a human received a 'not helpful' score from the judge across all its evaluations, highlighting a critical gap between the judge's precision and its accuracy. AI
IMPACT Highlights the challenges in using LLMs for automated evaluation and the need for careful calibration against human judgment.
RANK_REASON The item details the results of an evaluation of an LLM judge against a human-rated dataset, including statistical analysis and thresholds. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
