English(EN) Can LLM Annotators Match Human Consistency? Benchmarking Human Agreement and Optimizing Multi-Agent with Contrastive Knowledge

新的基准ANT-A和框架ANT-OR提高了LLM标注的一致性

作者 PulseAugur 编辑部 · [1 个来源] · 2026-10-26 04:00

研究人员推出了ANT-A基准测试，以解决当前LLM标注基准测试的局限性，这些基准测试通常未能考虑到人类标注者细微且有时不同的判断。这个新的基准测试包括特定任务的指南以及七个不同数据集上每个实例的五个独立人类标签。为了改进LLM标注系统，他们还提出了ANT-OR，一个多智能体框架，该框架使用从反馈中得出的对比决策规则来优化提示，与基线方法相比，一致性准确性提高了4.5个百分点。 AI

影响这项研究可能带来更强大、更适应的LLM标注系统，更好地反映现实世界中的人类判断。

排序理由该集群描述了一篇介绍LLM标注基准测试和框架的新学术论文。[lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.CL 阅读 →

AI 生成摘要 · Google Gemini · 来自 1 个来源。我们如何撰写摘要 →

本文如何被排名

Signal score: 15 / 100; Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket: Tool; 该集群描述了一篇介绍LLM标注基准测试和框架的新学术论文。[lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration: Single-source cluster; Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics: paper, product; Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance: High; Clearly on-topic for AI-industry coverage.
Story freshness: Breaking (< 6h); Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

完整方法见我们的编辑标准。

报道来源 [1]

arXiv cs.CL TIER_1 English(EN) · Xiaopeng Ke, Zhixin Wang, Xuebo Liu, Xinyu Ma, Xing Zheng, Bowen Song, Weiqiang Wang, Min Zhang · 2026-10-26 04:00

大型语言模型标注员能否媲美人为一致性？基准测试人类协议并优化多代理对比知识

arXiv:2610.32415v1 Announce Type: new Abstract: LLM-based annotation has advanced rapidly, yet existing benchmarks remain misaligned with real-world practice. Actual annotation often involves lengthy, task-specific guidelines, and different annotators may produce divergent but re…

本文如何被排名

报道来源 [1]

大型语言模型标注员能否媲美人为一致性？基准测试人类协议并优化多代理对比知识

相关实体

相关话题