用作自动化评估系统中裁判的大型语言模型会表现出不一致性，导致结果不可靠。采样温度、模型版本漂移、提示模糊和决胜机制等因素都会导致这种可变性。为缓解这些问题，开发人员可以实施诸如将温度设置为零、固定特定模型版本和提示版本、对多次运行的分数取平均值以及将分数量化为更粗糙的网格等策略。此外，一个重大挑战是立场偏见，即 LLM 裁判倾向于偏爱第一个呈现的答案，这会扭曲评估指标。解决此问题需要以两种呈现顺序运行成对比较，以衡量分歧并确定偏见的程度。 AI
影响 不一致的 LLM 裁判和立场偏见可能导致模型评估不可靠，需要稳健的方法来进行准确的性能评估。
排序理由 该集群讨论了用于提高基于 LLM 的评估系统的可重复性和公平性的研究结果和方法。
AI 生成摘要 · Google Gemini · 来自 3 个来源。 我们如何撰写摘要 →
<p>Same outputs, same judge, two runs, two scores. The gate flickered red then green on a branch with zero code changes, and that flapping cost me more trust than any real regression.</p> <h2> The flap </h2> <p>I had a faithfulness gate on merge: judge scores every case, the mean…
<p>The flap</p> <p>I had a faithfulness gate on merge: judge scores every case, the mean has to clear 0.80. One Tuesday it failed at 0.79. I re-ran the identical job, no code change, no prompt change, and it passed at 0.82. Ran it a third time: 0.80 exactly. Nothing in the repo h…
<p>If your pairwise judge sees answer A before answer B, it tends to prefer A. If you never swap the order, every win-rate you report is contaminated by which slot you happened to put each answer in.<br /> The first time I actually measured this, I did not believe the number. I h…