Revisiting Vul-RAG: Reproducibility and Replicability of RAG-based Vulnerability Detection with Open-Weight Models
A new study revisits the Vul-RAG framework for detecting software vulnerabilities using retrieval-augmented generation (RAG) with open-weight models. Researchers found that while the framework's results are reproducible in a local setting, performance plateaus around 0.30 pairwise accuracy, even with more advanced models. This suggests that simply increasing model capacity does not significantly improve vulnerability detection effectiveness, highlighting trade-offs between detection accuracy, model capabilities, and scale. AI
IMPACT Confirms that current open-weight models struggle to surpass a specific performance threshold for vulnerability detection, indicating a need for architectural or knowledge-integration improvements beyond raw scale.