近期研究对大型语言模型（LLMs）在用作评估AI生成文本的评判者时的可靠性提出了担忧。研究表明，LLM评判者可能过度依赖评分标准本身，即使不审阅生成的内容也能给出可预测的分数。此外，当输入文本或评估标准发生变化时，这些模型有时未能调整其判断，这表明其推理能力不够稳健。这项工作强调了深入研究LLM驱动的评估系统的机制和潜在偏见的必要性，尤其是在多语言环境中。 AI
影响 引发了对LLM生成文本的自动化评估指标有效性的担忧，可能影响模型开发和基准测试。
排序理由 该集群包含多篇在arXiv上发表的学术论文，讨论了LLM作为评判者（LLM-as-a-Judge）系统的的方法论和可靠性。
AI 生成摘要 · Google Gemini · 来自 4 个来源。 我们如何撰写摘要 →
完整方法见我们的编辑标准。
arXiv:2609.02942v1 Announce Type: cross Abstract: LLM-as-a-Judge pipelines are increasingly used to evaluate AI-generated text, based on the assumption that judgments arise from reasoning over candidate responses with respect to a rubric. We show that this assumption warrants fur…
arXiv:2503.17039v3 Announce Type: replace-cross Abstract: Automatic text summarization relies on automatic evaluation to quickly determine the quality of summarization models via automatic metrics and LLM-as-a-Judge models. However, these techniques require meta-evaluation to ens…
arXiv:2609.01604v1 Announce Type: new Abstract: LLM-based evaluators of natural language generation (NLG) quality are widely deployed as scoring tools and as automated training signals, yet the internal procedure by which they assign a rating remains poorly understood. We investi…
LLM-based evaluators of natural language generation (NLG) quality are widely deployed as scoring tools and as automated training signals, yet the internal procedure by which they assign a rating remains poorly understood. We investigate this procedure mechanistically through an e…