A new leaderboard from Vidraft evaluates how well AI agents can distinguish between legitimate and malicious shell commands, particularly when the commands are obfuscated using methods like base64 encoding. The evaluation found that while most models are adept at blocking genuinely risky commands, their performance varies significantly in correctly passing benign commands, which is crucial for agent functionality. Kimi K3.1 Pro and GPT 6.1 Terra demonstrated strong performance in both identifying malicious commands and maintaining high pass rates for legitimate ones, even under adversarial conditions like persuasive or threatening text. AI
IMPACT Highlights the critical need for AI agents to accurately differentiate between benign and malicious commands, even when obfuscated, to ensure reliable and safe operation.
RANK_REASON The item details a new evaluation benchmark and leaderboard for AI safety, specifically focusing on the ability of AI agents to detect obfuscated malicious shell commands. [lever_c_demoted from research: ic=1 ai=1.0]
- DeepSeek V3.2
- DeepSeek V4.1 Flash
- FINAL-Bench
- glm-5.3(-flash)
- GPT 6.1 Terra
- Kimi K3.1 Pro
- Lazarus Group
- Llama 3.3-70B
- Microsoft
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →