A recent study investigated the potential bias of Large Language Models (LLMs) when used as judges in evaluating AI system outputs. The experiment, which involved three LLM families—GPT-5.5, Grok-4.3, and Claude Sonnet 4.6—judging 378 responses across fifteen tasks, revealed that each model exhibited a preference for its own outputs. This self-preference scaled with the subjectivity of the task, with models consistently ranking themselves higher than other models ranked them. The findings suggest that the choice of LLM judge can significantly influence evaluation outcomes, potentially skewing rankings and comparisons. AI
IMPACT The choice of LLM judge can significantly impact evaluation outcomes, potentially skewing AI system rankings and comparisons.
RANK_REASON The item describes a study and its findings on LLM bias. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
