English(EN) Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation

研究发现：提示词设计导致大语言模型创新性评估不稳定

作者 PulseAugur 编辑部 · [1 个来源] · 2026-10-15 04:00

一篇新近发表在arXiv上的研究揭示了大型语言模型（LLMs）在评估想法创新性方面存在显著的不稳定性。研究人员发现，为LLM评估者设计的提示词的微小改动会极大地改变其评估结果，有时甚至导致准确率低于随机猜测水平。即使在呈现相同的想法对时，仅仅告知LLM先前审稿人的意见就可能使其在超过一半的案例中翻转其判决。研究还指出，检索增强和更大的推理预算几乎没有带来改进，而专门设计的创新性评估器表现甚至不如一个基础的提示词LLM基线。这些发现对自动化构思系统报告的创新性提升提出了质疑，并强调了需要更稳健的评估方法。 AI

影响引发了对AI生成想法的可靠性以及用于评估这些想法的方法的担忧。

排序理由学术论文，详细介绍了对LLM评估方法进行的受控研究。[lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.CL 阅读 →

AI 生成摘要 · Google Gemini · 来自 1 个来源。我们如何撰写摘要 →

本文如何被排名

Signal score: 12 / 100; Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket: Tool; 学术论文，详细介绍了对LLM评估方法进行的受控研究。[lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration: Single-source cluster; Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics: paper, other; Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance: High; Clearly on-topic for AI-industry coverage.
Story freshness: Breaking (< 6h); Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

完整方法见我们的编辑标准。

报道来源 [1]

arXiv cs.CL TIER_1 English(EN) · Noy Sternlicht, Simra Shahid, Peter Jansen, Daniel S. Weld, Pao Siangliulue, Tom Hope · 2026-10-15 04:00

旧思想，新问题：基于LLM的新颖性评估的不稳定性

arXiv:2610.02022v2 Announce Type: replace Abstract: AI-based ideation systems are often evaluated on the novelty of their ideas, and that judgment is increasingly delegated to large language models. Such judges are typically built ad hoc and validated, if at all, on human-authore…

本文如何被排名

报道来源 [1]

旧思想，新问题：基于LLM的新颖性评估的不稳定性

相关实体

相关话题