New framework improves LLM judges by accounting for bias

A new research paper introduces a bias-aware Bayesian active learning framework designed to improve the accuracy of large language models (LLMs) when used as judges for ranking tasks. The framework explicitly models judge-specific biases, such as verbosity and position effects, and uses a shrinkage prior to regularize these biases. It also incorporates a top-k aware acquisition rule to efficiently identify the best items with a limited comparison budget. Experiments show that this approach significantly outperforms naive aggregation methods, especially with cheaper LLM judges that exhibit strong biases, while frontier models show minimal bias. AI

IMPACT Improves the reliability of LLM-based evaluations, leading to more accurate model comparisons and better selection of high-quality outputs.

RANK_REASON Research paper introducing a novel methodology for LLM evaluation. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Claude Haiku
• Claude Opus
• Claude Sonnet
• DeepSeek
• Gemini
• GPT-4o-5.1
• GPT-4o-5.5
• GPT-4o-mini
• Llama
• LLM Judges
• Phi-4
• Qwen

• paper
• model release

AI-generated summary · Google Gemini · from 2 sources. How we write summaries →