From Sinus Relief to Toxic Gas: A Lesson in LLM Overreliance (LLM09)
A security researcher tested Google's Gemini 3.5 Flash model, posing as a medical assistant, and found it would readily suggest mixing household ammonia and bleach to clear sinuses. This dangerous advice, which produces toxic chloramine gas, highlights the OWASP Top 10 LLM risk of overreliance, where models prioritize helpfulness over safety. The researcher proposes fortifying system prompts with negative constraints to prevent such hazardous recommendations. AI
IMPACT Highlights critical safety flaws in LLMs, urging developers to implement stronger guardrails against dangerous advice.