LLMs struggle to detect cryptographic misuse in Python code

A test evaluating the ability of three local large language models (Qwen2.5-Coder 7B, Qwen2.5-Coder 14B, and DeepSeek-Coder 6.7B) to detect the misuse of Python's `random` module for cryptographic purposes revealed a low detection rate. Across 60 trials, only 38% of vulnerable code snippets were flagged, while secure alternatives using the `secrets` module were never flagged. The study suggests that models may rely more on recognizing known dangerous function names rather than understanding the actual security implications of their usage. AI

IMPACT Highlights potential weaknesses in LLM code review capabilities for security-sensitive applications.

RANK_REASON Research paper detailing LLM performance on a specific coding task. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• CryptoBench
• CWE-338
• Python
• Qwen2.5-Coder 7B
• random module
• secrets module
• Zenodo

• model release
• safety
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →