PulseAugur
EN
LIVE 01:47:02
한국어(KO) base64로 감춘 위험 명령, AI 안전 관문은 알아볼까: 난독화 셸 안전도 리더보드 20개 모델 실측

AI agents tested on detecting obfuscated malicious shell commands · 1 source tracked

A new leaderboard from Vidraft evaluates how well AI agents can distinguish between legitimate and malicious shell commands, particularly when the commands are obfuscated using methods like base64 encoding. The evaluation found that while most models are adept at blocking genuinely risky commands, their performance varies significantly in correctly passing benign commands, which is crucial for agent functionality. Kimi K3.1 Pro and GPT 6.1 Terra demonstrated strong performance in both identifying malicious commands and maintaining high pass rates for legitimate ones, even under adversarial conditions like persuasive or threatening text. AI

IMPACT Highlights the critical need for AI agents to accurately differentiate between benign and malicious commands, even when obfuscated, to ensure reliable and safe operation.

RANK_REASON The item details a new evaluation benchmark and leaderboard for AI safety, specifically focusing on the ability of AI agents to detect obfuscated malicious shell commands. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →

AI agents tested on detecting obfuscated malicious shell commands · 1 source tracked

How we ranked this

Signal score
31 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Tool
The item details a new evaluation benchmark and leaderboard for AI safety, specifically focusing on the ability of AI agents to detect obfuscated malicious shell commands. [lever_c_demoted from res…
Source corroboration
Single-source cluster
Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics
safety, model release
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
Breaking (< 6h)
Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

Full methodology in our editorial standards.

COVERAGE [1]

  1. dev.to — LLM tag TIER_1 한국어(KO) · Vidraft_lab ·

    Can AI Safety Gateways Detect Base64-Obfuscated Malicious Commands? Empirical Study of 20 Models on the Obfuscated Shell Safety Leaderboard

    <h2> TL;DR </h2> <ul> <li>위험한 셸 명령을 base64나 hex로 감싸면 AI 에이전트의 안전장치가 알아볼까요? 비드래프트가 운영하는 「난독화 셸 안전도 리더보드」는 이 질문을 비공개 평가 세트 221개 명령, 3회 다수결, AI 심판 없음으로 잽니다.</li> <li>균형정확도 1위는 GPT-6.1 Terra(99.5), 오픈 모델 1위는 Kimi K3.1 Pro(97.9), 3위는 DeepSeek V4.1 Flash(95.2)입니다. 상위권은 위장된 명령도 풀어서 의도를 …