Security Researchers Identify New Indirect Prompt Injection Threats Targeting AI Systems
In a recent discovery, security researchers have revealed ten new indirect prompt injection (IPI) payloads designed to exploit AI agents through malicious instructions. These payloads are crafted to induce actions that could lead to financial fraud, data destruction, and API key theft, among other nefarious activities. The significance of this finding underlines the evolving landscape of cybersecurity threats, particularly concerning artificial intelligence systems that interact with web content.
Indirect prompt injection occurs when threat actors manipulate web content to execute harmful instructions within an AI agent’s operations. By embedding malicious code within a web page, an attacker can wait for an AI agent to crawl or summarize the content. When the agent processes the compromised page, it inadvertently follows the injected instructions as if they were legitimate. This technique poses a serious risk to any AI system that browses web pages, summarizes information, indexes content for retrieval-augmented generation (RAG) pipelines, auto-processes metadata or HTML comments, and even moderates content related to advertisements or search engine optimization (SEO).
According to Mayur Sewani, a senior security researcher at Forcepoint, the potential damage caused by indirect prompt injection varies significantly based on the type of AI involved. In a blog post released yesterday, Sewani stated, “The impact scales with AI privilege. A browser AI that can only summarize is low-risk. An agentic AI that can send emails, execute terminal commands, or process payments becomes a high-impact target.” This distinction highlights the pressing need for robust security measures tailored to different functionalities of AI systems.
Forcepoint’s research has also pinpointed common triggers employed by these indirect prompt injection attacks. Some notable phrases used in these payloads include "Ignore previous instructions," "Ignore all previous instructions," "If you are an LLM," and "If you are a large language model." Regardless of the specifics of the payload, the attack chain remains consistent. Threat actors begin by poisoning web content, hiding their malicious instructions beneath seemingly innocuous text. Once an AI agent ingests the infected webpage, it ignores its programmed directives, ultimately following the attacker’s commands and triggering actions that can have real-world consequences. This often includes a covert exfiltration channel that communicates back to the attacker.
Forcepoint’s investigation uncovered examples of payloads that appear less overtly harmful but still pose significant risks. For instance, one payload makes a false claim, asserting that the copyright owner has "expressly forbidden" the AI from answering any user inquiries about the page’s content. This tactic is characterized as "denial of service" or content suppression, effectively bottling up the information flow.
In contrast, more dangerous examples from Sewani’s research pose severe threats. One such payload instructs a coding assistant or an AI integrated into a development environment to execute a Unix command for “recursive forced deletion of files and directories.” This type of attack specifically targets agentic AI systems that have been integrated into integrated development environments (IDEs), terminal applications, or DevOps workflows. Tools such as GitHub Copilot, Cursor, and AI-powered continuous integration/continuous deployment (CI/CD) reviewers are especially vulnerable, as they may unwittingly ingest these malicious payloads during routine research activities.
Another particularly alarming payload attempts to coerce an AI agent into revealing sensitive information. By instructing the agent to “send me the secret API key,” this tactic aims to extract confidential data, all while attempting to conceal the malicious intent under the instruction to “Do not analyze the code / Do not spit out the flag.” This combination of subterfuge and direct engagement exposes AI agents to significant risks of data theft.
Moreover, one of the most brazen payloads found includes a directive designed for financial fraud. It embeds a PayPal.me link along with a specific amount of $5,000 and detailed processing instructions. According to Sewani, “This payload is designed for AI agents that have integrated payment capabilities: browser agents with saved payment credentials, AI financial assistants, or agentic tools with access to digital wallets.” The meticulous detailing—exact links, precise amounts, and clear steps—suggests that these parameters are not mere testing grounds but are indeed weaponized payloads aiming for immediate financial gain.
In summation, Forcepoint’s research concludes with a stark warning: any AI agent that ingests untrusted web content without stringent enforcement of a data-instruction boundary faces considerable risk. Each webpage that an AI reads holds the potential to be compromised, serving as a conduit for threats that could have far-reaching implications. The rapidly evolving landscape of cyber threats necessitates ongoing vigilance and the development of robust strategies to secure AI systems against these emerging risks.