New dataset tests how adversarial comments fool LLM code reviewers

A new dataset called CommentInject has been released to test how adversarial code comments can influence LLM code reviewers. The dataset contains 30 vulnerable Python functions across various CWE categories, with comments designed to mislead reviewers into believing the code is safe. Studies using models like Qwen2.5-Coder 7B and Deepseek-Coder 6.7B showed a significant drop in vulnerability detection when misleading comments were present, though the effect varied by model and comment strategy. AI

IMPACT Highlights potential vulnerabilities in LLM code review tools, necessitating improved adversarial robustness.

RANK_REASON Release of a new dataset for evaluating LLM safety/robustness in code review. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• codegemma 7B
• codellama 7B
• CommentInject
• CWE-89
• GitHub
• Hugging Face
• Ollama
• Python
• Qwen2.5-Coder 7B
• Sunny Chokshi
• Zenodo

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →