A new benchmark called SecLog-Bench has been developed to evaluate the capabilities of large language models (LLMs) in cybersecurity log reasoning. This benchmark addresses limitations in existing datasets by offering a larger scale, incorporating multi-source logs (audit, application, and network traffic), and providing detailed, step-by-step evidence chains for each sample. Evaluations of eight mainstream LLMs on SecLog-Bench revealed modest performance in attack-progress reasoning, with the top model achieving only 44.44% accuracy and 45.07% Macro-F1, indicating that current LLMs still struggle with reliable reasoning in this domain. AI
IMPACT This benchmark highlights current LLM limitations in cybersecurity reasoning, suggesting a need for further development in robust, evidence-based analysis for security applications.
RANK_REASON The item describes a new benchmark for evaluating LLMs in a specific research domain (cybersecurity log reasoning). [lever_c_demoted from research: ic=1 ai=1.0]
Read on Hugging Face Daily Papers →
- application logs
- Audit logs--a security tool for CPRs
- cybersecurity
- large language models
- Macro F1
- SecLog-Bench
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
