AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems
A new benchmark called AgentLeak has been developed to assess privacy risks in multi-agent Large Language Model (LLM) systems. Unlike previous benchmarks that only examined final outputs, AgentLeak analyzes internal communication channels between agents, such as inter-agent messages and shared memory. An evaluation using this benchmark across seven privacy-relevant pathways and 1,000 scenarios revealed that while multi-agent configurations can reduce leakage in final outputs, they introduce significant internal channel leakage, with inter-agent messages being a primary concern. The study highlights that standard output-only defenses are insufficient for securing multi-agent LLM systems. AI
IMPACT Highlights the need for new security measures beyond output monitoring for multi-agent LLM systems.