LLMs Show Uneven Repeatability in Security Audits, Benchmark Reveals

A new benchmark, Snyk VulnBench JS 1.0, has been developed to evaluate the repeatability of large language model (LLM) security reviews. The benchmark found that while LLM findings can vary significantly between runs, reference-matched findings show greater stability. The research suggests that combining agentic LLM security review with deterministic static application security testing (SAST) tools like Snyk Code offers a more robust approach than relying on either method alone. AI

IMPACT Highlights the need for hybrid approaches in AI-assisted code security, combining LLMs with traditional SAST tools for improved reliability.

RANK_REASON The cluster contains an academic paper detailing a new benchmark and its findings. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Claude
• Javascript
• Snyk Code
• Snyk VulnBench JS 1.0

• paper
• safety
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →