LLM agents patch security bugs, pass all tests, but still leave the vulnerability open [R]
A new benchmark, CVE-Bench, was developed to evaluate LLM agents' ability to patch security vulnerabilities in Python projects. Across 18 projects and 20 real-world CVEs, the best performing models achieved only a 50% success rate in fully patching vulnerabilities. Notably, even when models appeared to fix a bug and pass regression tests, the vulnerability often remained, highlighting a dangerous failure mode where the fix is indistinguishable from a correct one without hidden security tests. AI
![LLM agents patch security bugs, pass all tests, but still leave the vulnerability open [R]](https://preview.redd.it/g29hj3ndzt4h1.png?width=140&height=89&auto=webp&s=3a5e47d1059b68a9fd4964f32b4cd2f7b26d7658)
IMPACT LLM agents show significant limitations in reliably patching security vulnerabilities, indicating a need for more robust testing and development before deployment in security-critical applications.