一个名为BonaFide的新基准已被开发出来，用于评估用于评估大型语言模型推理的忠实度指标。该基准包含13个任务和10个模型的3,066个标记的思维链，揭示大多数现有的忠实度指标表现不佳，通常接近随机水平。这些指标还表现出偏差，并随着思维链的增长而退化，凸显了当前评估方法中的重大差距，以及对更可靠、更有效的替代方案的需求。 AI
影响 凸显了当前LLM评估中的关键局限性，在开发出更可靠的忠实度指标之前，可能会减缓其采用。
排序理由 该集群关注一篇介绍用于评估LLM忠实度指标的新基准的学术论文。
在 Hugging Face Daily Papers 阅读 →
AI 生成摘要 · Google Gemini · 来自 2 个来源。 我们如何撰写摘要 →
arXiv:2605.25052v1 Announce Type: new Abstract: Chains of thought (CoTs) have become central in interpreting and auditing behaviors of large language models. Yet growing evidence suggests that these traces often fail to faithfully represent the computations behind a model's predi…
Researchers created a benchmark with 3,066 labeled chains of thought examples across 13 tasks and 10 models to systematically evaluate faithfulness metrics, revealing that most metrics perform near randomly and have significant limitations in reliability and efficiency.