PulseAugur
中
实时 12:20:10
English(EN) From Self-Confidence to Statistical Accuracy: Scaling Human-in-the-Loop Evaluation with Direct and LLM-Assisted Auditing

新框架使用自我评估和LLM裁判来扩展LLM评估

作者 PulseAugur 编辑部 · [1 个来源] ·

本文介绍了一个统计框架，用于评估大型语言模型（LLMs）和多模态大型语言模型（MLLMs），该框架结合了直接人工审计和LLM辅助方法。它提出了用于直接审计的置信度分层采样，以提高准确性估计，即使在自我评估校准不准确的情况下也是如此。对于LLM辅助评估，本文使用敏感性和假阳性率来模拟裁判的表现，从而推导出误差校正估计器。在MMLU-Pro和MMMU-Pro上的实验表明，虽然LLM辅助评估可以提供效率，但其实际效益受到推理成本和应用特定条件的限制。 AI

影响 提供了一个新的统计框架来评估LLMs和MLLMs，有可能提高模型评估的效率和准确性。

排序理由 论文发表在arXiv上，详细介绍了LLM和MLLM评估的新统计框架。[lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.CL 阅读 →

AI 生成摘要 · Google Gemini · 来自 1 个来源。 我们如何撰写摘要 →

新框架使用自我评估和LLM裁判来扩展LLM评估

本文如何被排名

Signal score
9 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Tool
论文发表在arXiv上，详细介绍了LLM和MLLM评估的新统计框架。[lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration
Single-source cluster
Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics
paper, model release
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
Same-day
Cluster formed today. Ranking reflects the current source set at time of score.

完整方法见我们的编辑标准。

报道来源 [1]

  1. arXiv cs.CL TIER_1 English(EN) · Harine Choi (Texas A&M Transportation Institute, Texas A&M University System), Chenchen Kuai (Zachry Department of Civil and Environmental Engineering, Texas A&M University), Zihao Li (Department of Civil, Construction and Environmental Engineering, Marq… ·

    从自信到统计准确性：使用直接和LLM辅助审计来扩展人在回路中的评估

    arXiv:2610.35698v1 Announce Type: new Abstract: For large language models (LLMs) and multimodal large language models (MLLMs), model-reported self-confidence is often miscalibrated, yet it may still be useful for efficient evaluation. At the same time, LLM-as-a-judge offers a sca…

相关实体

相关话题