LLM judges show family bias in preference evaluations, study finds

A new arXiv paper investigates how the choice of Large Language Model (LLM) used for judging affects preference outcomes in pairwise comparisons. The study found that the LLM judge's own model family significantly influences its judgments, introducing a bias that is confounded with the quality of the candidates being evaluated. Researchers developed a corrected estimator to isolate this judge-family effect, revealing a consistent positive 'same-family lift' across four tested model families: Llama 3.1, Qwen 2.5, Gemma 2, and Yi 1.5. The study also identified position bias as another failure mode in LLM-as-judge setups, with a substantial percentage of pairwise outcomes reversing based on the order of presentation. AI

IMPACT Highlights potential biases in LLM evaluation methods, suggesting a need for more robust and unbiased judging systems for future model development.

RANK_REASON The cluster contains an academic paper detailing a new methodology and findings regarding LLM evaluation. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• Bradley-Terry model
• Gemma 2
• Llama 3.1
• LLM-as-Judge
• Qwen 2.5

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →