English(EN) Behind Aggregate Scores: What Systematically Varying Arena-Hard's LLM Judge Reveals

LLM 裁判的判决对指令的改变高度敏感

作者 PulseAugur 编辑部 · [1 个来源] · 2026-10-23 04:00

一篇新发表在 arXiv 上的研究调查了 LLM 裁判在评估 AI 模型响应方面的可靠性。研究人员发现，即使指令发生微小变化或相同设置重复运行，也可能显著改变单个判决结果，尽管聚合分数保持相似。这种敏感性凸显了当 LLM 生成的判决被用作偏好标签或评估单个响应时可能存在的问题，表明评估应包含提示级别的协议度量以及聚合分数。 AI

影响凸显了基于 LLM 的评估中潜在的不可靠性，影响了模型性能的评估和比较方式。

排序理由详细介绍 LLM 评估方法和发现的学术论文。[lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.CL 阅读 →

AI 生成摘要 · Google Gemini · 来自 1 个来源。我们如何撰写摘要 →

本文如何被排名

Signal score: 4 / 100; Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket: Tool; 详细介绍 LLM 评估方法和发现的学术论文。[lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration: Single-source cluster; Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics: paper, safety; Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance: High; Clearly on-topic for AI-industry coverage.
Story freshness: Same-day; Cluster formed today. Ranking reflects the current source set at time of score.

完整方法见我们的编辑标准。

报道来源 [1]

arXiv cs.CL TIER_1 English(EN) · Louise Yorke · 2026-10-23 04:00

Behind Aggregate Scores: What Systematically Varying Arena-Hard's LLM Judge Reveals

arXiv:2610.30512v1 Announce Type: new Abstract: LLM judges are used to determine which of two answers is better. Trusting these verdicts requires knowing how repeatable they are and how much they depend on the judge's instructions. We show that similar aggregate scores can concea…

本文如何被排名

报道来源 [1]

Behind Aggregate Scores: What Systematically Varying Arena-Hard's LLM Judge Reveals

相关实体

相关话题