New benchmark CREBench tests LLMs on cryptographic reverse engineering

Researchers have introduced CREBench, a new benchmark designed to evaluate the capabilities of large language models (LLMs) in the domain of cryptographic binary reverse engineering. The benchmark consists of 432 challenges derived from standard cryptographic algorithms and insecure key usage scenarios, with varying difficulty levels. In evaluations, GPT-5.4 demonstrated the strongest performance among eight frontier LLMs, achieving a score of 64.03 out of 100 and successfully recovering flags in 59% of challenges, though human experts still maintained a significant advantage with a score of 92.19. AI

IMPACT This benchmark could spur development of LLMs specialized for security tasks like vulnerability discovery and malware analysis.

RANK_REASON The cluster describes a new academic benchmark and evaluation of LLMs on a specific task, published on arXiv. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Baicheng Chen
• Capture the Flag
• CREBench
• cryptographic programs
• GPT-5.4
• large-language models
• malware analysis
• reverse engineering

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →