Brief

Researchers discovered a significant prompt injection vulnerability in the Perplexity Comet browser, allowing attackers to execute malicious instructions by hiding them within invisible elements on web pages. This indirect prompt injection attack, which requires no user interaction beyond asking the AI to summarize content, can lead to sensitive data exfiltration, including email addresses and one-time passwords. While Perplexity has issued fixes, the underlying architectural issue of AI models not distinguishing between content and instructions remains a broader concern for AI-enhanced applications processing external data. AI

UltraProbe, a new free AI security scanner, has been released by Ultra Lab to address the growing threat of prompt injection attacks on LLM applications. The tool offers two scanning modes: one that analyzes a system prompt for vulnerabilities in under five seconds, and another that scans a website's URL to detect risks associated with integrated AI chatbots. UltraProbe aims to provide accessible and comprehensive security testing for developers, covering major attack vectors identified by OWASP. AI

Prompt injection, a security risk where users manipulate AI models with malicious inputs, has become a significant operational concern. The author details their experiences with this threat, particularly within an ERP system, and analyzes the cost and effectiveness of various defense strategies. Initial methods like input validation and heuristic filtering proved insufficient due to high false positive rates and bypassability, while canary token approaches offered some success but were also vulnerable to sophisticated attacks. AI

Instead of blocking prompt injection attacks, the MIRAGE system uses a honeypot approach to deceive attackers. When a suspicious prompt is detected, MIRAGE feeds the attacker fabricated data and logs their actions, making them believe they are succeeding. This method aims to waste the attacker's resources and collect intelligence on their techniques, rather than alerting them to their detection. AI

AI red-teaming offers a structured approach for security teams to identify vulnerabilities in large language model applications. Key steps include defining the system's purpose, input/output capabilities, and potential adversaries to tailor testing. Prompt injection, both direct and indirect, is a primary attack vector to explore, alongside testing layered controls like content filters and output validation. AI

A security analysis highlights the risks associated with AI systems that interpret engineering blueprints, such as those developed at Skoltech. These systems, which use multimodal models to read and analyze architectural drawings and building codes, introduce new attack surfaces. Researchers warn of potential threats like steganographic prompt injection, where hidden instructions are embedded in blueprints, and data poisoning, which could lead to structurally unsound designs and catastrophic failures. AI

Prompt injection attacks, analogous to SQL injection for LLMs, pose a significant security risk by allowing malicious users to manipulate AI model behavior. These attacks can override system instructions, extract sensitive prompts, or exfiltrate data. Developers can defend against these threats using a multi-layered approach, starting with a fast, keyword-based blocklist to catch obvious attempts, followed by a more sophisticated method using a separate, isolated LLM to classify potentially malicious inputs. AI

A security researcher developed a method to defend AI agents against prompt injection and malformed data attacks. This approach aims to enhance the robustness and safety of AI systems when interacting with potentially malicious inputs. AI

Researchers have developed a new method called 'exemplification' to exploit privacy vulnerabilities in black-box chatbot environments. This technique allows attackers to hijack an agent's intended task by crafting seemingly benign external content that redirects the chatbot to execute malicious objectives. The study demonstrates a data-exfiltration chain by combining prompt injection, instruction steering, and web-tool invocation, highlighting a feasible privacy-leakage path in deployed chatbot agents. AI

Fireworks AI has developed a new feature called 'safe_tokenization' to prevent prompt injection attacks in large language models. This technique ensures that user input, which can contain malicious control tokens, is treated as data rather than code by the model. By distinguishing between user-provided text and the model's internal control tokens, safe_tokenization maintains the integrity of prompt structures, preventing unauthorized alterations to model behavior. AI