English(EN)Candidate supply and answer selection shape the value of LLM judging in multi-agent systems
研究表明，多智能体系统中的LLM评判面临可靠性问题
作者PulseAugur 编辑部·[9 个来源]·
多篇研究论文探讨了在多智能体系统和评估智能体工具调用中使用大型语言模型（LLM）作为评判者的局限性和潜在改进。一项研究引入了AgentAuditor，它使用推理树来比多数投票更有效地聚合智能体输出，并引入了Anti-Consensus Preference Optimization来减少从众偏见。另一篇论文《LLM作为评判者并非神谕》（LLM-as-a-Judge Is Not an Oracle）强调了在使用LLM评判者的自改进智能体管道中存在的重大故障，并提出了一种名为PROCTOR的确定性验证层来缓解奖励破解和偏见等问题。进一步的研究调查了LLM评判者与人类判断分布对齐的挑战，提出了NAPHA等方法来改进软标签预测，并研究了候选者供应和答案选择如何影响多智能体系统中LLM评判的价值，指出正确答案通常存在但可能被忽略。最后，一项关于多智能体辩论（MAD）的研究表明，基于共识的协议可能会引入向下的偏见，并降低主观评估的人类对齐度，建议采用对称协议作为潜在的改进方法。
AI
arXiv:2602.09341v2 Announce Type: replace Abstract: Multi-agent systems (MAS) can substantially extend the reasoning capacity of large language models (LLMs). Most MAS frameworks aggregate agent outputs via simple majority voting, discarding the evidential structure of reasoning …
arXiv:2609.02246v1 Announce Type: new Abstract: Self-improving agent pipelines have a problem at their center. An optimizer rewrites prompts to score higher, and the score comes from a judge that is itself an LLM. That judge has the last word on whether the system is getting bett…
arXiv cs.CL
TIER_1English(EN)·Sebastian Steindl, Nikos Voskarides, Alberto Gasparin, Diego Marcheggiani·
arXiv:2609.01073v1 Announce Type: new Abstract: The LLM-as-a-judge (LLMaJ) framework offers a cost-effective and reproducible solution for automatic evaluation. However, current evaluation practices typically compare LLMaJ judgments against aggregated ground-truth labels, overloo…
arXiv:2608.25937v2 Announce Type: replace Abstract: Multi-agent systems (MAS) sometimes already have the potential to answer correctly, but still report a wrong answer. Explaining this outcome is difficult because generation, communication and final answer-selection rules usually…
arXiv cs.CL
TIER_1English(EN)·Minsoo Song, Chanwoo Kim, Sugyeong Eo, Chanjun Park·
arXiv:2608.30373v1 Announce Type: new Abstract: Multi-Agent Debate (MAD) has been widely adopted to improve LLM-based evaluation by prompting multiple agents to negotiate and reach a consensus. However, for subjective rubric-based scoring, inter-agent agreement does not guarantee…
Multi-Agent Debate (MAD) has been widely adopted to improve LLM-based evaluation by prompting multiple agents to negotiate and reach a consensus. However, for subjective rubric-based scoring, inter-agent agreement does not guarantee alignment with human judgments. In this paper, …
AgentJudgeBench reveals that LLM judges face structural reliability limits on dependency-driven agentic tool-calling workflows, with alignment degrading by difficulty and ground-truth exposure yielding mixed effects.
Multi-agent systems (MAS) sometimes already have the potential to answer correctly, but still report a wrong answer. Explaining this outcome is difficult because generation, communication and final answer-selection rules usually change simultaneously. We conceptualize multi-agent…
dev.to — LLM tag
TIER_1English(EN)·Ama Senevirathne·
<p>When an agent regression judge says “accept,” the interesting question is often not the score. It is whether another judge agrees, whether either judge showed its work, and what happens when they do not. JuryTrace is a small Python tool for that boundary: it runs two configure…
