New framework audits prompt sensitivity in LLM vulnerability detection

Researchers have developed PromptAudit, a framework to assess how prompt variations affect Large Language Models (LLMs) used for vulnerability detection. Their study, which tested five prompting strategies on five open-weight models using 1,000 CVEs across 16 programming languages, revealed that standard chain-of-thought prompting yielded the best results. The findings indicate that prompt sensitivity is a critical factor in LLM performance for vulnerability detection and should be a key consideration during evaluation and deployment. AI

影响 Highlights the critical role of prompt engineering in ensuring the reliability and accuracy of LLMs for security applications.

排序理由 The cluster contains an academic paper detailing a new framework and experimental results for evaluating LLM performance. [lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.AI 阅读 →

• chain-of-thought prompting
• few-shot prompting
• LLM
• PromptAudit

• 论文
• 安全

AI 生成摘要 · Google Gemini · 来自 1 个来源。 我们如何撰写摘要 →