A user has developed a tool to benchmark AI code reviewers against real-world bugs and CVEs. The tool feeds known vulnerabilities and their fixes to various AI models, scoring their ability to detect them. Initial results show significant performance gaps between models, with some consistently identifying specific bug classes while others miss them or hallucinate issues. Notably, the latest Claude model reportedly refused to review security code. AI
IMPACT Highlights significant disparities in AI code review capabilities, impacting software development security and efficiency.
RANK_REASON User-developed tool for benchmarking AI code reviewers.
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
