Two new benchmarks, JavaVulBench and RustMizan, have been released to evaluate the capabilities of large language models in detecting software vulnerabilities. JavaVulBench focuses on Java methods and includes over 1,740 Common Vulnerabilities and Exposures (CVEs), offering multiple realistic split strategies for testing. RustMizan addresses Rust vulnerabilities with compilable code and a mutation framework to test for contamination and robustness. Both benchmarks aim to provide more realistic and comprehensive evaluations than previous datasets, which often used small code snippets and lacked contamination awareness. AI
IMPACT These benchmarks will enable more rigorous evaluation of LLMs for code security, potentially accelerating the adoption of AI in software development security.
RANK_REASON The cluster contains two research papers introducing new benchmarks for evaluating AI models on software vulnerability detection.
- Claude Sonnet 4
- CodeBERT
- CodeLlama 13B
- Common Weakness Enumeration
- DeepSeek-Coder-1.3B
- DeepSeek-Coder-V2
- DeepSeek-V3
- GPT-4.1 mini
- GPT-4o
- GraphCodeBERT
- Java
- JavaVulBench
- Ollama
- OpenRouter
- Qwen 2.5 Coder 14B
- Qwen-2.5-Coder-7B
- Rust
- RustMizan
- UniXcoder
AI-generated summary · Google Gemini · from 2 sources. How we write summaries →
