PulseAugur
实时 14:23:05
中文(ZH) 前沿 AI 安全法规:AI 公司员工参考指南

AI代理面临新的提示注入和后门攻击

研究人员正在开发新的方法来攻击和防御用于软件逆向工程和网络安全的人工智能代理。一种方法使用遗传算法将恶意提示注入AI代理,导致它们误解代码并绕过检测系统。其他研究侧重于检测和混淆这些提示注入攻击,以及防御嵌入代理工作流程中持久控制的多步木马攻击。此外,一个名为CVE-Factory的框架自动化了用于训练和评估代码安全代理的可执行漏洞任务的创建,展示了Qwen3-32B等模型显著的改进。 AI

影响 AI代理新的攻击向量和防御机制凸显了AI驱动工具中关键的安全漏洞。

排序理由 多篇研究论文详细介绍了针对安全领域AI代理的新型攻击和防御方法。

在 METR (Model Evaluation & Threat Research) 阅读 →

AI 生成摘要 · Google Gemini · 来自 104 个来源。 我们如何撰写摘要 →

AI代理面临新的提示注入和后门攻击

报道来源 [104]

  1. arXiv cs.AI TIER_1 English(EN) · Jun He, Deying Yu ·

    主权保障边界:面向Agentic基础设施的证书绑定准入

    arXiv:2606.11632v1 Announce Type: cross Abstract: Agentic infrastructure introduces a critical control-plane authorization problem: non-deterministic reasoning systems can propose high-stakes mutations to production resources, yet existing security mechanisms -- such as identity …

  2. arXiv cs.MA (Multiagent) TIER_1 English(EN) · Deying Yu ·

    主权保障边界:面向代理基础设施的证书绑定准入

    Agentic infrastructure introduces a critical control-plane authorization problem: non-deterministic reasoning systems can propose high-stakes mutations to production resources, yet existing security mechanisms -- such as identity and access management (IAM), policy engines, conse…

  3. arXiv cs.AI TIER_1 English(EN) · Bijaya Dangol ·

    从隐私到工作流完整性:自主代理互操作性中的通信图元数据

    arXiv:2606.07150v1 Announce Type: cross Abstract: Agent-interoperability protocols such as A2A and MCP standardize what agents say to one another, but assume address-based transport over HTTP(S). Such transports protect message content, increasingly with end-to-end encryption. Wh…

  4. arXiv cs.AI TIER_1 English(EN) · Thamilvendhan Munirathinam ·

    代理会自行回避吗?衡量LLM代理对带内访问拒绝信号的合规性

    arXiv:2606.06460v1 Announce Type: cross Abstract: As autonomous LLM agents increasingly hold real credentials and operate infrastructure without a human in the loop, operators have no standard way to tell an agent that a resource is off-limits. Access controls either let the agen…

  5. arXiv cs.AI TIER_1 English(EN) · Hanna Foerster, Tom Blanchard, Kristina Nikoli\'c, Ilia Shumailov, Cheng Zhang, Robert Mullins, Nicolas Papernot, Florian Tram\`er, Yiren Zhao ·

    CaMeLs 也能使用电脑:电脑使用代理的系统级安全

    arXiv:2601.09923v3 Announce Type: replace Abstract: AI agents are vulnerable to prompt injection attacks, where malicious content hijacks agent behavior. Among proposed defenses, architectural isolation provides the strongest guarantees by strictly separating trusted task plannin…

  6. arXiv cs.AI TIER_1 English(EN) · Charlie Summers, Eugene Wu ·

    数据流控制:AI代理的数据安全策略

    arXiv:2606.05679v1 Announce Type: cross Abstract: Agents increasingly generate SQL, orchestrate pipelines, and automate data analysis on behalf of users. While recent work improves query correctness, correctness is not safety. A query may be semantically valid yet violate regulat…

  7. arXiv cs.AI TIER_1 English(EN) · Rufat Asadli, Benjamin Hoffman, Ioannis Protogeros, Laurent Vanbever ·

    评估用于计算机网络的Agentic配置修复

    arXiv:2606.06212v1 Announce Type: new Abstract: Misconfigurations in computer networks remain a major source of critical Internet outages. Research is turning to Large Language Models (LLMs) to automate the complex, error-prone task of network configuration. However, even state-o…

  8. arXiv cs.MA (Multiagent) TIER_1 English(EN) · Bijaya Dangol ·

    从隐私到工作流完整性:自主代理互操作性中的通信图元数据

    Agent-interoperability protocols such as A2A and MCP standardize what agents say to one another, but assume address-based transport over HTTP(S). Such transports protect message content, increasingly with end-to-end encryption. What they leave in the clear is the communication gr…

  9. arXiv cs.CL TIER_1 English(EN) · Nicholas Saban ·

    领域条件下的前沿计算机使用代理安全:一个793集浏览器基准测试、一个编码领域交叉引用以及近期红队测试的可复现性审计

    arXiv:2606.05233v1 Announce Type: cross Abstract: Recent computer-using-agent (CUA) red-teaming papers report prompt-injection attack success rates (ASR) of 42-98%, but these headline numbers cluster on retired models and on the most-vulnerable model in each paper's panel. We ask…

  10. arXiv cs.AI TIER_1 English(EN) · Thamilvendhan Munirathinam ·

    代理会自行回避吗?衡量LLM代理对带内访问拒绝信号的遵从性

    As autonomous LLM agents increasingly hold real credentials and operate infrastructure without a human in the loop, operators have no standard way to tell an agent that a resource is off-limits. Access controls either let the agent in (it has valid credentials) or hard-fail it (i…

  11. arXiv cs.AI TIER_1 English(EN) · Laurent Vanbever ·

    评估用于计算机网络的Agentic配置修复

    Misconfigurations in computer networks remain a major source of critical Internet outages. Research is turning to Large Language Models (LLMs) to automate the complex, error-prone task of network configuration. However, even state-of-the-art models fail to resolve misconfiguratio…

  12. Hugging Face Daily Papers TIER_1 English(EN) ·

    数据流控制:AI代理的数据安全策略

    Agents increasingly generate SQL, orchestrate pipelines, and automate data analysis on behalf of users. While recent work improves query correctness, correctness is not safety. A query may be semantically valid yet violate regulatory, privacy, or business constraints that govern …

  13. arXiv cs.AI TIER_1 English(EN) · Pritam Dash, Tongyu Ge, Aditi Jain, Tanmay Shah, Zhiwei Shang ·

    从不可信输入到可信记忆:LLM Agent 中记忆投毒攻击的系统性研究

    arXiv:2606.04329v1 Announce Type: cross Abstract: Memory is a core component of AI agents, enabling them to accumulate knowledge across interactions and improve performance. However, persistent memory introduces the risk of memory poisoning, where a single adversarial memory writ…

  14. arXiv cs.CL TIER_1 English(EN) · Aradhye Agarwal, Gurdit Siyan, Yash Pandya, Joykirat Singh, Akshay Nambi, Ahmed Awadallah ·

    学习何时行动或拒绝:保护代理推理模型以安全地进行多步工具使用

    arXiv:2603.03205v2 Announce Type: replace Abstract: Agentic language models operate in a fundamentally different safety regime than chat models: they must plan, call tools, and execute long-horizon actions where a single misstep, such as accessing files or entering credentials, c…

  15. arXiv cs.AI TIER_1 English(EN) · Yiqi Wang, Jiaqi Zhang, Taotao Cai, Zirui Liu, Qingqiang Sun, Zequn Sun, Zhangkai Wu, Mingkai Zhang, Yanming Zhu ·

    从Agent轨迹到信任:LLM Agent中的证据追踪与执行溯源

    arXiv:2606.04990v1 Announce Type: cross Abstract: Large language model (LLM)-based agents increasingly solve complex tasks by interacting with external tools, retrieval systems, memory modules, environments, and other agents. These capabilities expand agent autonomy, but also mak…

  16. arXiv cs.AI TIER_1 English(EN) · Tianneng Shi, Robin Rheem, Dongwei Jiang, Mona Wang, Francisco De La Riega, Zhun Wang, Jingzhi Jiang, Alexander Cheung, Sean Tai, Jonah Cha, Jianhong Tu, Gabriel Han, Chenguang Wang, Jingxuan He, Wenbo Guo, Dawn Song ·

    CyberGym-E2E:AI代理端到端网络安全能力的可扩展真实世界基准

    arXiv:2606.04460v1 Announce Type: cross Abstract: AI has the potential to transform cybersecurity by enabling systems that can autonomously detect, analyze, and remediate software vulnerabilities. However, existing cybersecurity evaluations of AI systems are limited in scale or s…

  17. arXiv cs.AI TIER_1 English(EN) · Yuanbo Xie, Tianyun Liu, Yingjie Zhang, Suchen Liu, Yulin Li, Liya Su, Tingwen Liu ·

    如果提示注入从未消失?探索代理系统中的跨会话存储提示注入

    arXiv:2606.04425v1 Announce Type: cross Abstract: Modern agentic systems transform LLMs from session-bounded assistants into stateful systems that persist and evolve shared world state across sessions through memories, filesystems, tools, and other long-lived contextual artifacts…

  18. arXiv cs.AI TIER_1 English(EN) · Juan Figuera ·

    公证代理:接收者证明的 AI 代理行动机密收据

    arXiv:2606.04193v1 Announce Type: cross Abstract: Current AI agent observability is structurally compromised: the entity producing the activity log is the same entity whose activity is being logged. A compromised or buggy agent can omit, alter, or fabricate its own traces, and th…

  19. arXiv cs.AI TIER_1 English(EN) · Yanming Zhu ·

    从Agent追踪到信任:LLM Agent中的证据追踪与执行溯源

    Large language model (LLM)-based agents increasingly solve complex tasks by interacting with external tools, retrieval systems, memory modules, environments, and other agents. These capabilities expand agent autonomy, but also make agent behavior harder to verify, debug, and audi…

  20. arXiv cs.AI TIER_1 English(EN) · Jinliang Xu ·

    OpenAgenet/OAN: 信任治理的代理身份与发现的技术架构

    arXiv:2606.03163v1 Announce Type: cross Abstract: This paper describes the technical architecture of OpenAgenet / OAN. OAN is a protocol-neutral trust layer for open Agent interconnection. It specifies the role architecture, identity objects, registration workflow, Root-governed …

  21. arXiv cs.AI TIER_1 English(EN) · Eliot Krzysztof Jones, Mateusz Dziemian, Matt Fredrikson, J Zico Kolter ·

    AI Agent网络安全拒绝的新框架

    arXiv:2606.02644v1 Announce Type: cross Abstract: Agentic scaffolds have dramatically improved LLM performance on complex, long-horizon tasks, yielding both broad benefits and amplified risks in domains like cybersecurity. Existing benchmarks for AI agents in cybersecurity focus …

  22. arXiv cs.AI TIER_1 English(EN) · Jinliang Xu ·

    OpenAgenet/OAN: 开放式可信智能体互联基础设施

    arXiv:2606.03161v1 Announce Type: cross Abstract: OpenAgenet, abbreviated as OAN, is an open infrastructure project for trusted Agent interconnection. It addresses a problem that becomes visible when Agents move from isolated applications into open, multi-operator networks: befor…

  23. arXiv cs.MA (Multiagent) TIER_1 English(EN) · Jinliang Xu ·

    OpenAgenet/OAN: 信任治理的代理身份和发现的技术架构

    This paper describes the technical architecture of OpenAgenet / OAN. OAN is a protocol-neutral trust layer for open Agent interconnection. It specifies the role architecture, identity objects, registration workflow, Root-governed lifecycle, Root-verified package model, authorizat…

  24. arXiv cs.MA (Multiagent) TIER_1 English(EN) · Jinliang Xu ·

    OpenAgenet/OAN: 信任治理的代理身份与发现技术架构

    This paper describes the technical architecture of OpenAgenet / OAN. OAN is a protocol-neutral trust layer for open Agent interconnection. It specifies the role architecture, identity objects, registration workflow, Root-governed lifecycle, Root-verified package model, authorizat…

  25. arXiv cs.MA (Multiagent) TIER_1 English(EN) · Jinliang Xu ·

    OpenAgenet/OAN: 可信代理互联的开放基础设施

    OpenAgenet, abbreviated as OAN, is an open infrastructure project for trusted Agent interconnection. It addresses a problem that becomes visible when Agents move from isolated applications into open, multi-operator networks: before an Agent can safely discover, select, and invoke…

  26. arXiv cs.MA (Multiagent) TIER_1 English(EN) · Jinliang Xu ·

    OpenAgenet/OAN: 可信代理互联的开放基础设施

    OpenAgenet, abbreviated as OAN, is an open infrastructure project for trusted Agent interconnection. It addresses a problem that becomes visible when Agents move from isolated applications into open, multi-operator networks: before an Agent can safely discover, select, and invoke…

  27. arXiv cs.AI TIER_1 English(EN) · Riddhi Mohan Sharma ·

    Ethical Hyper-Velocity (EHV):面向Agentic AI系统的硬件根零信任运行时强制执行架构

    arXiv:2605.17909v2 Announce Type: replace Abstract: As autonomous agentic systems scale across regulated critical infrastructures, the lack of mechanistic, hardware-rooted enforcement for high-frequency policy updates presents a fundamental safety gap. We present Ethical Hyper-Ve…

  28. arXiv cs.AI TIER_1 English(EN) · Ismail Hossain, Sai Puppala, Zhuoran Lu, Sajedul Talukder, Nan Jiang ·

    开放代理技能生态系统中的安全风险检测与验证基准测试

    arXiv:2606.00925v1 Announce Type: cross Abstract: Open agent platforms allow community contributors to publish reusable skills that agents can invoke at runtime. This extensibility also creates a supply-chain risk: malicious contributors can hide harmful behavior inside skills th…

  29. arXiv cs.AI TIER_1 English(EN) · Florian Holzbauer, David Schmidt, Gabriel Gegenhuber, Sebastian Schrittwieser, Johanna Ullrich ·

    上下文很重要:Agent Skill生态系统的存储库感知安全分析

    arXiv:2603.16572v2 Announce Type: replace-cross Abstract: Agent skills extend local AI agents, such as Claude Code and OpenClaw, with additional functionality. Their growing popularity has led to dedicated marketplaces resembling mobile app stores, as well as automated scanners t…

  30. arXiv cs.AI TIER_1 English(EN) · Yi Liu, Zhihao Chen, Yanjun Zhang, Gelei Deng, Yuekang Li, Jianting Ning, Leo Yu Zhang ·

    “不要对用户提及”:检测和理解恶意代理技能

    arXiv:2602.06547v3 Announce Type: replace-cross Abstract: LLM-based coding agents increasingly rely on third-party extensions called skills, which bundle natural language instructions and helper scripts that execute with full user privileges. Community registries have emerged to …

  31. arXiv cs.AI TIER_1 English(EN) · Jeremy Tien, Abishek Anand, Yu-Rou Tuan, Yuchen Shen, J. Zico Kolter, Aran Nayebi ·

    ROGUE: 普通电脑使用中出现的失调代理行为

    arXiv:2606.00341v1 Announce Type: cross Abstract: As AI agents are increasingly deployed in real personal and corporate settings (email accounts, development workflows, company databases, etc.), safety considerations surrounding these agents become paramount. Although much work h…

  32. arXiv cs.CL TIER_1 English(EN) · Soham Roy, Sarthakbrata Halder, Arya Bharaty, Vaibhav Bhaskar, Yash Sinha, Dhruv Kumar, Srikant Panda, Murari Mandal ·

    “我强烈怀疑这个网站是骗局”:在无防御的自主网络代理中对个人身份信息泄露和检测进行基准测试

    arXiv:2606.00497v1 Announce Type: cross Abstract: Deceptive web content, widely instantiated across the internet and commonly known as \textit{social-engineering attacks}, manipulates autonomous web agents into submitting users' personally identifiable information (PII) to attack…

  33. arXiv cs.CL TIER_1 English(EN) · Yunhao Feng, Yifan Ding, Xiaohu Du, Ming Wen, Xinhao Deng, Yanming Guo, Yuxiang Xie, Baihui Zheng, Yingshui Tan, Yige Li, Yutao Wu, Yixu Wang, Kerui Cao, Wenke Huang, Xingjun Ma, Yu-Gang Jiang ·

    BraveGuard:从开放世界威胁到更安全的计算机使用代理

    arXiv:2606.01166v1 Announce Type: cross Abstract: Computer-use agents extend language models from text generation to sustained interaction with files, terminals, browsers, and external tools. This shift creates safety risks that are difficult to detect from isolated prompts or fi…

  34. arXiv cs.AI TIER_1 English(EN) · Yoshinari Fujinuma, Varun Gangal, Traian Rebedea, Makesh Narasimhan Sreedhar, Prasoon Varshney, Rebecca Qian, Anand Kannappan ·

    针对基于终端的代理的技能注入攻击的防御和赋能因素

    arXiv:2606.01567v1 Announce Type: cross Abstract: Large language model (LLM) agents increasingly rely on reusable skills i.e. documents describing task-specific procedures. However, this introduces a new attack surface for agents to manage. We study two complementary directions f…

  35. arXiv cs.AI TIER_1 English(EN) · Hao Cheng, Changtao Miao, Tianle Song, Yin Wu, He Liu, Erjia Xiao, Junchi Chen, Xiaoyu Shi, Yichi Wang, Jing Yang, Taowen Wang, Jinhao Duan, Mengshu Sun, Peiyan Dong, Xuan Shen, Yang Cao, Renjing Xu, Kaidi Xu, Jindong Gu, Bo Zhang, Jize Zhang, Chenhao Li… ·

    SeClaw:用于评估自主代理的面向规范的安全任务合成

    arXiv:2606.02302v1 Announce Type: cross Abstract: Autonomous LLM agents increasingly operate in stateful environments where they access tools, files, memory, and external services. While such capabilities enable complex real-world workflows, they also introduce security risks tha…

  36. Hugging Face Daily Papers TIER_1 English(EN) ·

    BraveGuard:从开放世界威胁到更安全的计算机使用代理

    BraveGuard is a self-evolving defense framework that trains guard models using open-world threat signals and realistic agent trajectories to improve safety detection in computer-use agents.

  37. arXiv cs.AI TIER_1 English(EN) · Chao Shen ·

    SeClaw:用于评估自主代理的驱动式安全任务合成

    Autonomous LLM agents increasingly operate in stateful environments where they access tools, files, memory, and external services. While such capabilities enable complex real-world workflows, they also introduce security risks that are difficult to capture with existing evaluatio…

  38. arXiv cs.AI TIER_1 English(EN) · Brian Crawford, Patrick McClure ·

    Investigating Detection and Obfuscation of Prompt Injection Attacks Against Software Reverse Engineering AI Agents

    arXiv:2605.30677v1 Announce Type: cross Abstract: Agentic software reverse engineering systems are vulnerable to prompt injection attacks placed into the source code of executable binary files. This research demonstrates defensive tactics for detecting the presences of prompt inj…

  39. arXiv cs.AI TIER_1 English(EN) · Xianzhen Luo, Jingyuan Zhang, Shiqi Zhou, Jinyang Huang, Chuan Xiao, Qingfu Zhu, Zhiyuan Ma, Xing Yue, Yang Yue, Wencong Zeng, Wanxiang Che ·

    CVE-Factory:为代码安全漏洞扩展专家级代理任务

    arXiv:2602.03012v3 Announce Type: replace-cross Abstract: Evaluating and improving the security capabilities of code agents requires high-quality, executable vulnerability tasks. However, existing works rely on costly, unscalable manual reproduction and suffer from outdated data …

  40. arXiv cs.AI TIER_1 English(EN) · Brian Crawford, Justin Phillips, Patrick McClure ·

    自动攻击软件逆向工程AI代理

    arXiv:2605.30667v1 Announce Type: cross Abstract: Software tools for reverse engineering executable binary files, such as Ghidra, enable malware analysts to safely conduct robust static analysis without having access to original source code. Coupled with the analytic power of lar…

  41. arXiv cs.AI TIER_1 English(EN) · Jiejun Tan, Zhicheng Dou, Xinyu Yang, Yuyang Hu, Yiruo Cheng, Xiaoxi Li, Ji-Rong Wen ·

    从提示注入到持久控制:防御Agentic Harness免受特洛伊木马后门的侵害

    arXiv:2605.31042v1 Announce Type: cross Abstract: LLM agents are evolving from conversational chatbots to operational tools in real-world workspaces. In local agentic harnesses, an LLM can read and write files, call tools, and reuse workspace state across sessions. While such cap…

  42. arXiv cs.CL TIER_1 English(EN) · Ji-Rong Wen ·

    从提示注入到持久控制:防御Agentic Harness免受特洛伊木马后门攻击

    LLM agents are evolving from conversational chatbots to operational tools in real-world workspaces. In local agentic harnesses, an LLM can read and write files, call tools, and reuse workspace state across sessions. While such capabilities enhance utility, they also expose a new …

  43. arXiv cs.AI TIER_1 English(EN) · Suliu Qin, Haomin Zhuang, Yujun Zhou, Yufei Han, Xiangliang Zhang ·

    AIRGuard:通过运行时权限控制守护代理行为

    arXiv:2605.28914v1 Announce Type: cross Abstract: Tool-using language agents turn model decisions into external side effects: they read files, run scripts, call APIs, send messages, and invoke Model Context Protocol tools. This makes agent attacks different from jailbreaks. The h…

  44. arXiv cs.AI TIER_1 English(EN) · Dongrui Liu, Yu Li, Zhonghao Yang, Peng Wang, Guanxu Chen, Yuejin Xie, Qinghua Mao, Wanying Qu, Yanxu Zhu, Tianyi Zhou, Leitao Yuan, Zhijie Zheng, Qihao Lin, Yimin Wang, Haoyu Luo, Shuai Shao, Chen Qian, Qingyu Liu, Ling Tang, Ruiyang Qin, Qihan Ren, Jun… ·

    AgentDoG 1.5:AI Agent安全与保障的轻量级且可扩展的对齐框架

    arXiv:2605.29801v1 Announce Type: new Abstract: Modern open-world agents such as OpenClaw exhibit powerful cross-environment execution capabilities yet introduce broad new safety risk sources. Meanwhile, advanced frontier AI models drastically lower attack barriers, rendering cur…

  45. arXiv cs.AI TIER_1 English(EN) · Galip Tolga Erdem ·

    固定易受攻击目标面前的 AI 攻击者有多可靠?一项 400 次运行的 LLM 入侵测试一致性实证研究

    arXiv:2605.30096v1 Announce Type: cross Abstract: Large language models (LLMs) can autonomously conduct multi-stage cyber attacks, but the consistency of their offensive behavior under repeated trials remains unstudied. This work presents the first large-scale empirical measureme…

  46. arXiv cs.AI TIER_1 Svenska(SV) · Yunhao Feng, Yifan Ding, Yingshui Tan, Boren Zheng, Yanming Guo, Xiaolong Li, Kun Zhai, Yishan Li, Wenke Huang ·

    SkillTrojan:针对基于技能的代理系统的后门攻击

    arXiv:2604.06811v2 Announce Type: replace-cross Abstract: Skill-based agent systems tackle complex tasks by composing reusable skills, improving modularity and scalability while introducing a largely unexamined security attack surface. We propose SkillTrojan, a backdoor attack th…

  47. Hugging Face Daily Papers TIER_1 English(EN) ·

    从提示注入到持久控制:防御Agentic Harness免受特洛伊木马后门的侵害

    Multi-step trojan attacks in local LLM agents can bypass existing defenses by embedding malicious prompts across multiple operations, requiring new detection methods like DASGuard for effective protection.

  48. arXiv cs.AI TIER_1 English(EN) · Galip Tolga Erdem ·

    固定易受攻击目标下 AI 攻击者的可靠性如何?一项 400 次运行的 LLM 入侵测试一致性实证研究

    Large language models (LLMs) can autonomously conduct multi-stage cyber attacks, but the consistency of their offensive behavior under repeated trials remains unstudied. This work presents the first large-scale empirical measurement of LLM attack consistency: 400 autonomous penet…

  49. arXiv cs.CL TIER_1 English(EN) · Xia Hu ·

    AgentDoG 1.5: AI Agent 安全与可靠性的轻量级且可扩展的对齐框架

    Modern open-world agents such as OpenClaw exhibit powerful cross-environment execution capabilities yet introduce broad new safety risk sources. Meanwhile, advanced frontier AI models drastically lower attack barriers, rendering current agent alignment frameworks inadequate for r…

  50. arXiv cs.AI TIER_1 Svenska(SV) · Chang Jin, An Wang, Zeming Wei, Kai Wang, Biaojie Zeng, Qiaosheng Zhang, Chao Yang, Jingjing Qu, Xia Hu, Xingcheng Xu ·

    SkillSafetyBench:评估技能面向攻击面下的代理安全

    arXiv:2605.12015v2 Announce Type: replace-cross Abstract: Reusable skills are becoming a common interface for extending large language model agents, packaging procedural guidance with access to files, tools, memory, and execution environments. However, this modularity introduces …

  51. arXiv cs.LG TIER_1 English(EN) · Meghana Bhange, Ulrich A\"ivodji, Elliot Creager ·

    测试时集体行动:基于代理的扰动以纠正算法危害

    arXiv:2605.27689v1 Announce Type: new Abstract: When machine learning systems under-perform for particular subgroups, affected users typically have no way to correct these disparities without relying on platform-level fixes. Existing approaches to algorithmic fairness rely on pro…

  52. arXiv cs.AI TIER_1 English(EN) · Yaoyu Zhao, Yichen Xu, Oliver Bra\v{c}evac, Cao Nguyen Pham, Frank Zhengqing Wu, Martin Odersky ·

    LACUNA:安全代理作为递归程序漏洞

    arXiv:2605.28617v1 Announce Type: new Abstract: LLM agents increasingly act by writing code, yet a split persists between the runtime that drives the agent and the code the model writes. The runtime owns the loop, context, and control flow, and the model has little say over any o…

  53. arXiv cs.CL TIER_1 English(EN) · Jiaqian Li, Yanshu Li, Boxuan Zhang, Ruixiang Tang, Kuan-Hao Huang ·

    TRACES:通过轨迹-状态建模实现多轮 LLM 代理的主动安全审计

    arXiv:2605.27690v1 Announce Type: new Abstract: LLM agents increasingly operate through multi-turn tool use and environment interaction, where safety risks often emerge from intermediate steps long before they surface in the final outcome. Reactive auditing is therefore insuffici…

  54. Hugging Face Daily Papers TIER_1 English(EN) ·

    AgentDoG 1.5: AI Agent 安全与可靠性的轻量级且可扩展的对齐框架

    A lightweight and scalable agent safety alignment framework is proposed to address emerging threats from advanced AI models, featuring taxonomy-guided training with minimal samples and efficient deployment in real-world scenarios.

  55. arXiv cs.AI TIER_1 English(EN) · Martin Odersky ·

    LACUNA:安全代理作为递归程序漏洞

    LLM agents increasingly act by writing code, yet a split persists between the runtime that drives the agent and the code the model writes. The runtime owns the loop, context, and control flow, and the model has little say over any of them. Letting model-written code shape the run…

  56. arXiv cs.AI TIER_1 English(EN) · Changyue Jiang, Wenqi Zhang, Xudong Pan, Geng Hong, Min Yang ·

    三思而后行:通过思维纠正增强智能体行为安全性

    arXiv:2505.11063v3 Announce Type: replace Abstract: LLM-based agents solve complex tasks through iterative reasoning, tool use, and environment interaction, where each intermediate thought directly shapes subsequent actions. Small deviations in these thoughts can therefore propag…

  57. arXiv cs.AI TIER_1 English(EN) · Yige Li, Yunhao Feng, Jun Sun ·

    职位:人工智能安全需要有效的可控性

    arXiv:2605.27117v1 Announce Type: new Abstract: AI safety is still largely framed as alignment: training models to follow human preferences, safety policies, and normative constraints. That framing has improved the behavior of modern language models, but aligned behavior does not…

  58. arXiv cs.AI TIER_1 English(EN) · Yinghan Hou, Zongyou Yang, Zaihu Pang, Xiujun Ma ·

    SkillSieve:一种用于检测恶意AI代理技能的层级式分类框架

    arXiv:2604.06550v2 Announce Type: replace-cross Abstract: OpenClaw's ClawHub marketplace hosts tens of thousands of community-contributed agent skills (49,592 in our 2026-04-04 snapshot), and recent audits report that 13-26% contain security vulnerabilities. Regex scanners miss o…

  59. Hugging Face Daily Papers TIER_1 English(EN) ·

    LACUNA:安全代理作为递归程序漏洞

    LACUNA is a programming model that enables LLM agents to write code that shapes the runtime while maintaining safety through type checking and controlled execution.

  60. arXiv cs.AI TIER_1 English(EN) · Jun Sun ·

    职位:人工智能安全需要有效的可控性

    AI safety is still largely framed as alignment: training models to follow human preferences, safety policies, and normative constraints. That framing has improved the behavior of modern language models, but aligned behavior does not by itself guarantee that a deployed agent can b…

  61. arXiv cs.AI TIER_1 English(EN) · Jingwei Sun, Jianing Zhu, Yuanyi Li, Tongliang Liu, Xia HU, Bo Han ·

    AgentHijack:基准测试计算机使用代理对常见环境损坏的鲁棒性

    arXiv:2605.25707v1 Announce Type: new Abstract: Autonomous computer use agents that powered by multimodal large language models (MLLMs) are emerging as capable assistants for completing complex digital workflows. However, real-world execution environments are far from ideal: pop-…

  62. arXiv cs.AI TIER_1 English(EN) · Jinhu Qi, Muzhi Li, Jiahong Liu, Yuqin Shu, Dianzhi Yu, Shicheng Ma, Wenqian Cui, Yiyang Zhao, Yiyi Chen, Ruoxi Jiang, Irwin King, Zenglin Xu ·

    迈向可信的代理AI:安全、鲁棒性、隐私和系统安全综合调查

    arXiv:2605.23989v1 Announce Type: new Abstract: Agentic AI systems -- Large Language Models (LLMs) augmented with planning, tool use, memory, and long-horizon interactions -- can execute complex tasks autonomously, but their multi-step trajectories introduce new failure modes tha…

  63. arXiv cs.AI TIER_1 English(EN) · Bo Han ·

    AgentHijack:基准测试计算机使用代理对常见环境损坏的鲁棒性

    Autonomous computer use agents that powered by multimodal large language models (MLLMs) are emerging as capable assistants for completing complex digital workflows. However, real-world execution environments are far from ideal: pop-ups, resolution changes, and competing applicati…

  64. arXiv cs.AI TIER_1 English(EN) · Pepijn Cobben, Xuanqiang Angelo Huang, Thao Amelia Pham, Isabel Dahlgren, Terry Jingchen Zhang, Zhijing Jin ·

    GT-HarmBench:通过博弈论的视角对 AI 安全风险进行基准测试

    arXiv:2602.12316v2 Announce Type: replace Abstract: Frontier AI systems are increasingly capable and deployed in high-stakes multi-agent environments. However, existing AI safety benchmarks largely evaluate single agents, leaving multi-agent risks such as coordination failure and…

  65. arXiv cs.LG TIER_1 English(EN) · Jonathan N\"other, Adish Singla, Goran Radanovic ·

    MaMa:一种用于设计安全智能体系统的博弈论方法

    arXiv:2602.04431v2 Announce Type: replace Abstract: LLM-based multi-agent systems have demonstrated impressive capabilities, but they also introduce significant safety risks when individual agents fail or behave adversarially. In this work, we study the automated design of agenti…

  66. Hugging Face Daily Papers TIER_1 English(EN) ·

    AgentHijack:基准测试计算机使用代理对常见环境损坏的鲁棒性

    Computer-use agents powered by multimodal large language models face significant challenges in real-world environments due to dynamic disruptions, necessitating robustness evaluation and improved framework designs.

  67. arXiv cs.CL TIER_1 English(EN) · Piercosma Bisconti, Matteo Prandi, Federico Pierucci, Federico Sartore, Enrico Panai, Laura Caroli, Yue Zhu, Adam Leon Smith, Luca Nannini, Marcello Galisai, Susanna Cifani, Francesco Giarrusso, Marcantonio Bracale Syrnikov, Daniele Nardi ·

    煮蛙实验:面向Agent安全的多轮基准测试

    arXiv:2605.22643v1 Announce Type: new Abstract: Background. Traditional safety benchmarks for language models evaluate generated text: whether a model outputs toxic language, reproduces bias, or follows harmful instructions. When models are deployed as agents, the safety-relevant…

  68. Hugging Face Daily Papers TIER_1 English(EN) ·

    煮蛙实验:Agentic Safety 的多轮基准测试

    Background. Traditional safety benchmarks for language models evaluate generated text: whether a model outputs toxic language, reproduces bias, or follows harmful instructions. When models are deployed as agents, the safety-relevant object shifts from what the system says to what…

  69. arXiv cs.CL TIER_1 English(EN) · Daniele Nardi ·

    煮蛙实验:面向Agent安全的多轮基准测试

    Background. Traditional safety benchmarks for language models evaluate generated text: whether a model outputs toxic language, reproduces bias, or follows harmful instructions. When models are deployed as agents, the safety-relevant object shifts from what the system says to what…

  70. arXiv cs.AI TIER_1 English(EN) · Ahmad-Reza Sadeghi ·

    衡量安全性而不自欺欺人:为何评估智能体如此困难

    The benchmarks used to evaluate AI agents in security-critical roles suffer from crucial weaknesses. Building on recent empirical evidence, we characterize three core challenges that undermine security evaluations: benchmark vulnerabilities, temporal staleness, and runtime uncert…

  71. METR (Model Evaluation & Threat Research) TIER_1 中文(ZH) ·

    前沿人工智能安全法规:人工智能公司员工参考指南

    <p style="text-align: center;"><a class="button button-primary button-wide max-width-100" href="https://metr.org/frontier-ai-regulations.pdf">查看英文 PDF 版</a></p> <p>OpenAI、Google、Anthropic、xAI 等前沿 AI 开发者,以及部分中国 AI 开发者,已经需要遵守多项安全与安保义务。主要来源包括加州 SB 53、纽约 RAISE 法案,以及欧盟《人工智能法》中有关前沿 AI …

  72. METR (Model Evaluation & Threat Research) TIER_1 Español(ES) ·

    Frontier AI Safety Regulation: A Reference for Lab Personnel

    <p style="text-align: center;"><a class="button button-primary button-wide max-width-100" href="https://metr.org/frontier-ai-regulations.pdf">Ver como PDF</a></p> <p>Los desarrolladores de IA de frontera como OpenAI, Google, Anthropic, xAI y otros tienen obligaciones de seguridad…

  73. LessWrong (AI tag) TIER_1 English(EN) · Koby Lewis ·

    呼吁改进AI安全工具中的类型提示

    <p><span>Good type hints lead to code that is more </span><a href="https://link.springer.com/article/10.1007/s10664-013-9289-1" rel="noopener nofollow" target="_blank"><span>maintainable, easier to understand</span></a><span>, and with </span><a href="https://blog.acolyer.org/201…

  74. AWS Machine Learning Blog TIER_1 English(EN) · Bharathi Srinivasan ·

    使用 Amazon Bedrock AgentCore 网关中的策略和 Lambda 拦截器保护 AI 代理

    In this post, we use a lakehouse data agent to demonstrate how you can use Policy for deterministic access control and Lambda interceptors for dynamic validation. We then show how to combine Lambda interceptors and Policy to implement a geography-based access control which requir…

  75. Forbes — Innovation TIER_1 English(EN) · Anand Oswal, CommunityVoice ·

    Securing The Agentic Enterprise

    As organizations move beyond simple chatbots toward autonomous "compound systems" of agents, the traditional tech landscape has shifted.

  76. Forbes — Innovation TIER_1 English(EN) · Suman Sharma, Forbes Councils Member ·

    为什么消费级AI代理需要运行时安全,而不仅仅是治理

    Without the right controls, consumer-facing AI agents can expose organizations to regulatory violations, privacy breaches, eroded trust and reputational damage.

  77. Forbes — Innovation TIER_1 English(EN) · Lydia Zhang, Forbes Councils Member ·

    超越人工智能炒作:为何持续安全验证比以往任何时候都更重要

    Continuous testing matters because infrastructure changes constantly.

  78. Forbes — Innovation TIER_1 English(EN) · Robert Bobel, Forbes Councils Member ·

    AI 代理身份正在重新定义治理并扩大您的攻击面

    When an AI-driven process performs an action, accountability can span multiple teams, leaving no single point of responsibility.

  79. Forbes — Innovation TIER_1 English(EN) · Arti Raman, Forbes Councils Member ·

    消除危险的企业人工智能盲点

    Organizations are confronting the growing gap between AI hype and measurable business impact. This is exposing major blind spots in governance, usage visibility and operational oversight.

  80. Forbes — Innovation TIER_1 English(EN) · Tom Kellermann, Forbes Councils Member ·

    人工智能时代的网络警戒

    Threat detection and response must be accelerated across your entire digital estate.

  81. MarkTechPost TIER_1 English(EN) · Sana Hassan ·

    Microsoft Agent Governance Toolkit在安全AI Agent工具使用中的实现:包含策略、审批、审计日志和风险控制

    <p>In this tutorial, we build a governed AI-agent workflow using Microsoft’s Agent Governance Toolkit as the reference point. We create a Colab-ready implementation where agents do not directly execute tools; instead, every action first passes through a governance layer that chec…

  82. dev.to — Claude Code tag TIER_1 English(EN) · Marcus Rowe ·

    SymJack:一项让你的AI编码助手反戈一击的供应链攻击

    <p>Your AI coding agent just became an attack vector.</p> <p>That's the short version of what Adversa AI published this week. The research team disclosed a technique called SymJack — a symlink hijacking attack that turns AI coding assistants into supply chain attack delivery syst…

  83. dev.to — MCP tag TIER_1 English(EN) · Manveer Chawla ·

    面向CISO的AI代理治理与运行时合规框架

    <p>AI agents are now in production across healthcare, financial services, and critical SaaS systems. They mutate data, trigger workflows, and call external APIs on behalf of real users. These are autonomous actors, not the read-only recommendation engines that security teams alre…

  84. dev.to — MCP tag TIER_1 English(EN) · Baris Sozen ·

    法官或数学:自主代理结算的两种信任模型

    <p>When an AI agent settles a trade with no human watching, something has to make that trade trustworthy. There are two serious ways to do it, and they are not the same. One puts a judge in the loop. The other replaces the judge with math. Most of the current debate about "trust …

  85. Medium — MCP tag TIER_1 English(EN) · yunwei37 ·

    ACRFence:防止代理检查点恢复中的语义回滚攻击

    <div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@yunwei356/acrfence-preventing-semantic-rollback-attacks-in-agent-checkpoint-restore-b0d00f5e8b7b?source=rss------mcp-5"><img src="https://cdn-images-1.medium.com/max/1338/0*SgpvKg2tMyMYi5aC.pn…

  86. dev.to — MCP tag TIER_1 English(EN) · 云微 ·

    ACRFence:防止代理检查点恢复中的语义回滚攻击

    <p>AI agent frameworks are bringing checkpoint/restore, time travel, and rewind into everyday developer workflows. If an agent makes a mistake, it can go back to a checkpoint. If a user wants to explore another path, the agent can branch from an earlier state. This is useful for …

  87. dev.to — MCP tag TIER_1 English(EN) · Baris Sozen ·

    原子化是什么?隐藏在代理人结算中的星号

    <p>"Atomic" is having a moment. It is showing up in funding announcements, in launch threads, in agent-commerce pitch decks. This week a team raised <strong>$25M</strong> for an atomic OTC desk built on HTLCs and Bitcoin Taproot, with no custodian holding the assets mid-trade. Th…

  88. dev.to — Anthropic tag TIER_1 한국어(KO) · AI OpenFree ·

    Claude AI 威胁防护设计:内化价值观而非过滤器

    <h1> 클로드를 협박에 쓰지 못하게 막는 것과, 클로드가 스스로 협박하지 않도록 만드는 것은 전혀 다른 문제다 </h1> <p><em>앤트로픽이 '클로드'의 자기검열을 설계한 방식 — 그리고 왜 이것이 단순한 필터 이야기가 아닌가</em></p> <p><a class="article-body-image-wrapper" href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cform…

  89. Medium — AI coding tag TIER_1 English(EN) · Anna Jey ·

    AI 编码代理配置审计:如何在代理运行前发现风险代码库

    <div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/toward-next-ai/ai-coding-agent-configuration-audit-how-to-find-risky-repos-before-agents-run-98c0b34ed7b9?source=rss------ai_coding-5"><img src="https://cdn-images-1.medium.com/max/1672/1*vANJy…

  90. dev.to — LLM tag TIER_1 English(EN) · WonderLab ·

    Agent Series (13): Agent Security and Defense — Prompt Injection, Tool Abuse, and Data Leakage

    <h2> An Agent's Attack Surface Is Bigger Than You Think </h2> <p>A plain LLM application has one attack surface: user input → LLM output.</p> <p>Add tools to the mix, and it triples:<br /> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>User …

  91. Mastodon — fosstodon.org TIER_1 English(EN) · [email protected] ·

    无需攻击:用于 Agent Skills 规范违规的语义模糊测试 - LLM 驱动的代理可能默默删除文档、泄露凭证或传输

    "No Attack Required: Semantic Fuzzing for Specification Violations in Agent Skills" LLM-powered agents can silently delete documents, leak credentials, or transfer funds on a routine user request, not because the agent was attacked, but because the skill it invoked broke its own …

  92. dev.to — LLM tag TIER_1 English(EN) · Vaishnavi Gudur ·

    你的 AI 代理存在记忆问题:攻击者如何永久劫持它

    <p>Last week, I ran a simple experiment: I poisoned my own AI agent's memory with 3 lines of code. The result? The agent started leaking user data to an attacker-controlled endpoint — and it had no idea.</p> <h2> The Attack </h2> <p>Here's what memory poisoning looks like in prac…

  93. dev.to — LLM tag TIER_1 English(EN) · Falcons Edge ·

    AI Agent 安全:保护生产环境中的自主代理

    <p>Autonomous AI agents are moving from research labs into production environments at speed. Unlike chatbots that respond to single prompts, agents plan, reason, execute multi-step tasks, call external tools, and delegate sub-tasks to child agents. With each of these capabilities…

  94. dev.to — LLM tag TIER_1 English(EN) · Loïc Fontaine ·

    捕获AI代理传出消息中的提示注入(以及泄露的秘密)

    <p>AI agents now send email, post messages, and call tools on their own. We spend a<br /> lot of energy guarding the <strong>input</strong> — the user's prompt. We spend almost none on<br /> the <strong>output</strong>: what the agent is actually about to send.</p> <p>That's the …

  95. r/MachineLearning TIER_1 English(EN) · /u/TheAchraf99 ·

    [P] 免费AI代理安全评估 [P]

    <!-- SC_OFF --><div class="md"><p>Hey everyone,</p> <p>We’re building <strong>Antitech</strong>, a security layer for AI agents and LLM-powered workflows.</p> <p>We’re opening a small number of free early-access assessments for teams/builders working on AI agents.</p> <p>If you g…

  96. Mastodon — fosstodon.org TIER_1 English(EN) · [email protected] ·

    🚨 AI代理框架中的提示注入远程代码执行 在AI代理系统中通过提示注入实现远程代码执行的关键漏洞。主要发现

    🚨 Prompt Injection RCE in AI Agent Frameworks Critical vulnerabilities allowing remote code execution through prompt injection in AI agent systems. Key findings: • Which frameworks are affected • Attack chain to RCE • Real-world scenarios • Defensive recommendations Analysis → ht…

  97. Mastodon — fosstodon.org TIER_1 English(EN) · [email protected] ·

    🚨 提示注入漏洞允许在流行的 AI Agent 框架中执行远程代码 关键漏洞允许通过提示注入在 AI Agent 系统中执行远程代码

    🚨 Prompt Injection Flaws Enable RCE in Popular AI Agent Frameworks Critical vulnerability allowing remote code execution through prompt injection in AI agent systems. Full technical analysis → https:// cyber.murati.net # cybersecurity # infosec # AI # promptinjection # RCE # CVE

  98. Mastodon — fosstodon.org TIER_1 English(EN) · [email protected] ·

    计算器学科——AI辅助漏洞披露幻觉的分类法和预发送过滤器 - Stuart Thomas(独立研究者)的论文和工具

    The Calculator Discipline — A Taxonomy and Pre-Send Filter for AI-Assisted Vulnerability Disclosure Hallucinations - Paper and Tool by Stuart Thomas, independent Security Researcher # Infosec # LLM # AI https:// stuart-thomas.com/research/cal culator-discipline/

  99. dev.to — LLM tag TIER_1 English(EN) · Gian Paolo ·

    重大漏洞:数百万AI代理面临风险

    <h2> The Silent Threat: When Your AI Turns Against You </h2> <p>Your AI agent is sorting through a thousand new customer support emails, summarizing key issues and drafting responses. It has access to your company's private knowledge base, customer data, and internal APIs. It’s a…

  100. dev.to — LLM tag TIER_1 English(EN) · pueding ·

    Boiling the Frog Paper:多轮规范侵蚀与单提示代理安全

    <p><strong>What:</strong> The <strong>Boiling the Frog</strong> benchmark is a stateful multi-turn safety eval for tool-using AI agents — it walks a scenario from benign edits to risk-bearing actions and scores whether the agent accepts the escalated final turn.</p> <p><strong>Wh…

  101. dev.to — LLM tag TIER_1 English(EN) · Vaishnavi Gudur ·

    AgentThreatBench:首个 OWASP Agentic Top 10 安全基准测试

    <p>The AI safety community has a blind spot. We have excellent benchmarks for measuring whether an LLM will output harmful content (like toxicity or jailbreaks), and we have benchmarks for measuring whether an agent can successfully complete a task (like SWE-bench or WebArena).</…

  102. Mastodon — mastodon.social TIER_1 Français(FR) · [email protected] ·

    "Claw Patrol”:一款专为 AI 代理设计的开源防火墙。基本理念是可靠的——自主代理具有不同的攻击面

    "Claw Patrol" : un firewall open-source conçu spécifiquement pour les agents IA. L'idée de base est solide — les agents autonomes ont une surface d'attaque différente des apps classiques : appels d'outils, chaînes de prompts, accès externes. Avoir une couche de contrôle dédiée, c…

  103. r/Anthropic TIER_1 English(EN) · /u/theonejvo ·

    PolyRange:抗污染的进攻性AI基准测试,面向网络目标

    <!-- SC_OFF --><div class="md"><p>Author here. The short version of why I built this:</p> <p>Cyber-AI evaluation is converging on the same diagnosis from multiple labs. Anthropic's Claude Mythos system card this year: their cyber ranges &quot;lack many features often present in r…

  104. r/cursor TIER_2 English(EN) · /u/Few-Ad-1358 ·

    AI 编码代理的信任检查应在哪里进行?

    <!-- SC_OFF --><div class="md"><p>I’ve been using and studying AI coding agents more, and the part I keep getting stuck on is not whether they can write code. They obviously can. The harder question is where trust is supposed to enter the workflow. If an agent touches files outsi…