Researchers have introduced KaliBench, a new benchmark designed to evaluate Large Language Models (LLMs) on their ability to generate precise command-line interface (CLI) commands for cybersecurity tools. The benchmark includes 8,504 query-command pairs across 1,642 tools, focusing on the critical aspect of accurate command syntax and argument binding for cybersecurity operations. Initial evaluations show that current open-weight models struggle with this task, with none achieving over 42% exact-command accuracy. However, fine-tuning with KaliBench's verifiable rewards significantly improved an 8B model's performance, bringing it closer to that of a much larger 685B MoE model. AI
IMPACT Establishes a new standard for evaluating LLM performance in critical cybersecurity command-line tasks, potentially driving improvements in model accuracy and safety for tool invocation.
RANK_REASON The cluster describes a new benchmark and dataset for evaluating LLMs in a specific domain (cybersecurity CLI tool use), which falls under research.
Read on Hugging Face Daily Papers →
- KaliBench
- Kali Linux
- mixture of experts model
- natural-language--to--CLI
- open weight model
- LLMs
- RedSage-K-GRPO
- RISys-Lab
AI-generated summary · Google Gemini · from 2 sources. How we write summaries →