A recent study evaluated 16 open-source LLM code reviewers using 180 Python snippets across nine cryptographic misuse classes. The research found that relying solely on the true positive rate (TPR) for detecting vulnerabilities is misleading, as models that flagged almost everything, including secure code, appeared to perform best. The study introduced balanced accuracy, which considers both true positives and false positives, revealing that GPT OSS 20B achieved the highest score with a 93.7% TPR and a 3.3% FPR. AI
IMPACT Highlights the need for more robust evaluation metrics beyond simple detection rates for LLM code analysis tools.
RANK_REASON The item describes a research paper evaluating LLM code reviewers and introducing a new metric. [lever_c_demoted from research: ic=1 ai=1.0]
- CodeLlama 34B
- CodeLlama 7B
- CryptoBench
- devstral 24B
- GPT-OSS 120B
- GPT OSS 20B
- Institute of Electrical and Electronics Engineers
- Python
- Zenodo
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
