PulseAugur
EN
LIVE 19:58:11

LLM judges show biases and vulnerabilities in evaluation tasks · 4 sources tracked

By PulseAugur Editorial · [5 sources] ·

Recent research highlights significant biases and vulnerabilities in Large Language Model (LLM) judges, which are increasingly used for evaluating AI outputs. Studies reveal that these judges can be susceptible to model extraction attacks, where their judging capabilities are replicated with high accuracy across different evaluation protocols. Furthermore, LLM judges exhibit anchoring bias, where prior scores systematically influence subsequent judgments, compromising evaluation independence. Even when prompted to consider changes or disregard metadata, these biases persist, impacting the reliability of code evaluation and other tasks. AI

IMPACT Highlights critical flaws in LLM evaluation systems, necessitating improved methods for reliable AI assessment.

RANK_REASON Multiple academic papers published on arXiv detailing research into LLM judge biases and extraction.

Read on Hugging Face Daily Papers →

AI-generated summary · Google Gemini · from 5 sources. How we write summaries →

LLM judges show biases and vulnerabilities in evaluation tasks · 4 sources tracked

How we ranked this

Signal score
0 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Research
Multiple academic papers published on arXiv detailing research into LLM judge biases and extraction.
Source corroboration
5 independent sources
Strong cross-source corroboration — multiple independent publishers covered this within the clustering window.
Topics
paper, safety
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
78 days old
Aged out of breaking-news scoring windows; ranking reflects the durable signal from the full source set.
Coverage growth since scoring
+1 source(s) since last score
New sources have picked up this story since our last re-score. Score will update on the next scoring pass.

Full methodology in our editorial standards.

COVERAGE [5]

  1. arXiv cs.CL TIER_1 English(EN) · Chen Chen, Yaolin Chen, Xuehan Sun, Juan Lin, Xueluan Gong, Yuhang Zheng, Qian Wang, Kwok-Yan Lam ·

    JudgeStealer: Extracting LLM Judging Capabilities across Evaluation Protocols

    arXiv:2608.26982v1 Announce Type: new Abstract: Large language model (LLM) judges are increasingly used across various evaluation scenarios, making their judgment capabilities valuable intellectual property. However, black-box access exposes these capabilities to model extraction…

  2. Hugging Face Daily Papers TIER_1 English(EN) ·

    JudgeStealer: Extracting LLM Judging Capabilities across Evaluation Protocols

    Large language model (LLM) judges are increasingly used across various evaluation scenarios, making their judgment capabilities valuable intellectual property. However, black-box access exposes these capabilities to model extraction attacks. Existing extraction methods do not spe…

  3. arXiv cs.CL TIER_1 English(EN) · Ante Kapetanovic, Kemal Altwlkany, Andro Mercep, Tomislav Duricic, Emanuel Lacic ·

    Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence

    arXiv:2608.25869v1 Announce Type: new Abstract: Large language models (LLMs) increasingly assess generated content, giving rise to the LLM-as-a-Judge paradigm. These systems now score outputs, filter content, and gate iterative refinement in production pipelines, where each judgm…

  4. arXiv cs.AI TIER_1 English(EN) · Jianlin Chen, Wenhui Chen, Ziyao Lin, Chi Man Vong ·

    A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluation

    arXiv:2608.24419v1 Announce Type: new Abstract: LLM-as-a-judge evaluation is usually assessed by agreement and robustness to surface perturbations, but reliability does not establish construct validity. We formalize construct validity for an evaluator as a two-dimensional profile…

  5. arXiv cs.CL TIER_1 English(EN) · Jiwon Moon, Yerin Hwang, Dongryeol Lee, Taegwan Kang, Yongil Kim, Kyomin Jung ·

    Don't Judge Code by Its Cover: Exploring Biases in LLM Judges for Code Evaluation

    arXiv:2505.16222v2 Announce Type: replace Abstract: With the growing use of large language models(LLMs) as evaluators, their application has expanded to code evaluation tasks, where they assess the correctness of generated code without relying on reference implementations. While …

RELATED ENTITIES

RELATED TOPICS