新基准揭示代理场景中LLM作为裁判的评分噪声

报道来源 [3]

1. arXiv cs.CL TIER_1 English(EN) · Yangda Peng, Yunjia Qi, Hao Peng, Haotian Xia, Guanzhong He, Xintong Shi, Richeng Xuan, Songyuanyi Lu, Yixian Liu, Zhichao Hu, Yuhong Liu, Lei Hou, Bin Xu, Juanzi Li · 2026-06-30 04:00

   Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios?

   arXiv:2606.29920v1 Announce Type: new Abstract: Rubric-based scoring has become a widely used paradigm in model evaluation, typically with LLM-as-a-Judge (LaaJ) for rubric scoring. However, the reliability of LaaJ for rubric scoring remains underexplored. This concern is especial…

2. arXiv cs.CL TIER_1 English(EN) · Juanzi Li · 2026-06-29 07:57

   Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios?

   Rubric-based scoring has become a widely used paradigm in model evaluation, typically with LLM-as-a-Judge (LaaJ) for rubric scoring. However, the reliability of LaaJ for rubric scoring remains underexplored. This concern is especially pronounced in agentic scenarios, where long, …

3. dev.to — LLM tag TIER_1 English(EN) · Virginia Nyambura Mwega · 2026-07-01 15:54

   Evaluating Agents With an LLM-as-Judge Harness (Without Kidding Yourself About It)

   <p><strong>Key Takeaways</strong></p> <ul> <li>You can't unit-test a coach agent the way you test a pure function — the output is non-deterministic and "good" is a judgment call, not an assertion.</li> <li>An LLM-as-judge harness lets you grade a whole test set automatically agai…