New benchmark reveals LLM agents vulnerable to adaptive multi-turn attacks

Researchers have developed a new benchmark called Adaptive Adversaries to test the security of LLM agents against multi-turn, adaptive attacks. The benchmark revealed significant vulnerabilities, with attack success rates increasing from 0-1% in single-turn scenarios to 5.4-14.0% in multi-turn adaptive attacks. When using three frontier LLMs as attackers, the success rate was 1.4-2.2 times higher than with a single attacker, and the generated attacks showed low similarity to existing benchmarks. Claude Opus 4.6 and GPT-5.4 performed similarly overall, but exhibited different weaknesses across specific scenarios. AI

IMPACT Highlights critical security vulnerabilities in LLM agents, necessitating improved defenses against sophisticated, multi-turn attacks.

RANK_REASON The item is a research paper introducing a new benchmark for LLM agent security. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Adaptive Adversaries
• arXiv
• Claude Opus 4.6
• Gemini
• GPT-5.4
• GPT OSS 20B

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →