English(EN) Benchmark Scores Are Pipeline-Dependent: A Reliability Audit of Cybersecurity LLM Benchmarks

网络安全大语言模型基准因评估流程依赖性而不可靠

作者 PulseAugur 编辑部 · [1 个来源] · 2026-09-10 04:00

一项对网络安全大语言模型基准的新审计显示，基准分数高度依赖于所使用的评估流程，而非固定的数据集。研究人员识别出15种系统性故障模式，表明单一的评估流程选择可以将模型的得分改变80多个百分点，并显著改变其排名。即使是语义上相似的任务，由于评估约定不兼容，也可能产生不同的排名。该研究主张进行评估流程感知的审计，以确保模型评估的可靠性。 AI

影响强调了标准化评估方法学的必要性，以确保大语言模型性能指标的准确性和可比性。

排序理由该集群包含一篇详细介绍大语言模型基准可靠性研究结果的学术论文。[lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.AI 阅读 →

AI 生成摘要 · Google Gemini · 来自 1 个来源。我们如何撰写摘要 →

本文如何被排名

Signal score: 13 / 100; Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket: Tool; 该集群包含一篇详细介绍大语言模型基准可靠性研究结果的学术论文。[lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration: Single-source cluster; Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics: paper, other; Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance: High; Clearly on-topic for AI-industry coverage.
Story freshness: Breaking (< 6h); Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

完整方法见我们的编辑标准。

报道来源 [1]

arXiv cs.AI TIER_1 English(EN) · Aymene Berriche, Cathrine Shalby, Mohannad Alhanahnah, Yazan Boshmaf · 2026-09-10 04:00

基准分数依赖于管道：对网络安全大语言模型基准的可靠性审计

arXiv:2609.08765v1 Announce Type: cross Abstract: Large language model (LLM) benchmarks are often treated as fixed datasets with stable scores, yet their outcomes depend on configurable evaluation pipelines. We audit eight cybersecurity benchmarks across 10 proprietary, open-weig…

本文如何被排名

报道来源 [1]

基准分数依赖于管道：对网络安全大语言模型基准的可靠性审计

相关实体

相关话题