Mock tool calls fail to secure LLMs against untrusted inputs

Researchers explored using mock tool calls to quarantine untrusted inputs for large language models, hypothesizing it would improve robustness. Their experiments across seven models and three tasks revealed that this method generally did not enhance security and, in some cases, even increased attack success rates. The findings suggest a need for further evaluation of this limitation in deployed systems and the development of stronger instruction hierarchy training or new primitives for handling untrusted data. AI

IMPACT This research highlights a potential vulnerability in LLM security, suggesting that current methods for handling untrusted inputs may be insufficient and require further investigation.

RANK_REASON The cluster contains an academic paper detailing research findings on LLM security. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• GSM8K
• OpenAI

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →