New CryptoBench dataset tests LLM code review for cryptographic misuse

A new dataset called CryptoBench has been developed to evaluate the ability of large language models (LLMs) to detect cryptographic API misuse in code. The dataset consists of 54 Python snippets, including both vulnerable and secure examples, categorized by common weakness enumerations. Initial tests on seven open-source models revealed that while some models show high detection rates, many also incorrectly flag secure code as vulnerable, indicating a lack of discrimination. The dataset and associated analysis scripts are publicly available for further testing and research. AI

IMPACT This dataset could help improve the security auditing capabilities of AI code review tools, leading to more robust software development.

RANK_REASON The item describes a new dataset and benchmark for evaluating LLM code review capabilities, including a preprint paper. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• CodeLlama
• Common Weakness Enumeration
• CryptoBench
• cryptography
• DeepSeek Coder
• hashlib
• LLM
• Ollama
• pycryptodome
• PyJWT
• Qwen2.5 Coder
• Requests
• self-supervised learning
• Zenodo

• paper
• model release
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →