Attackers Can Manipulate AI Research Agents
Recent research from Cornell Tech has unveiled a pressing vulnerability regarding the manipulation of AI “deep-research” agents. These AI systems, specifically designed to peel back the layers of user queries and generate comprehensive reports, can be significantly influenced through simple alterations to user-generated content (UGC) on platforms like Reddit and Wikipedia. Attackers can achieve this with minimal effort by embedding as little as a 13-word snippet into existing threads or pages, which these agents may later reference as reliable information, including authentic advice, product recommendations, or even misleading scams.
The deep-research agents of concern include systems such as STORM, Co-STORM, and OmniThink. These multi-step systems operate by breaking down user inquiries into several interrelated sub-queries. They then embark on an extensive web search and compile their findings into detailed, citation-rich documents. Instead of relying on a static set of curated references, these systems are designed to absorb information directly from the open web, placing a heavy reliance on UGC platforms such as Reddit, Wikipedia, Quora, and various forums. Unfortunately, these platforms are not only high-ranking in search results but also considerably easy for adversaries to manipulate.
The research indicates that a substantial percentage of URLs fetched by these AI agents, ranging from 17% to 23%, are sourced from UGC. Notably, Reddit is highlighted as a primary target, with approximately half to two-thirds of all UGC-based URLs being pulled from its threads. This underscores the platform’s vulnerability, as it becomes a prime target for those adept at engaging in malicious digital activities. Within specific topic clusters—such as customer service queries or product recommendations—the same pages frequently emerge across various related queries. This suggests that compromising just a single thread can have far-reaching implications, affecting numerous related questions.
The study introduced a novel manipulation strategy named WARP (Web Agent Retrieval Poisoning). This approach exploits the high overlap and reoccurrence of URLs that deep-research agents frequently consult. The attackers first engage in reconnaissance by identifying widely referenced Reddit threads or Wikipedia entries that repeatedly surface on search results for targeted topics such as local business advice, financial investing, or online reviews. Armed with this information, they craft concise and carefully structured snippets—ranging from 80 to 120 words for comprehensive content or a mere 13 words for impactful search snippets—that promote fictional services or products.
These persuasive snippets are subtly injected as comments on Reddit threads, edits on Wikipedia pages, or replies in online forums. Once indexed by search engines, these snippets become integrated into the operation of deep-research agents, deceiving them into treating the manipulated information as legitimate.
Research findings illustrate a startling efficacy of this manipulation tactic. For instance, one poisoned snippet on Reddit can attain “mention” rates—where the fictitious product is actively recommended in responses—between 38% to 51% in open-source agents. Even when the manipulated paragraph constituted less than 4% of the overall content, agents still reiterated the misleading claims in approximately 30% to 53% of instances.
Concrete examples in the research reveal the potential ramifications of this vulnerability. A fabricated cryptocurrency, "BananaCoin," was unjustly elevated alongside well-known entities like Bitcoin and Ethereum within investment strategies after its name was integrated stealthily into comment threads. Likewise, an imaginary dating application, "SilverPath," found itself ranked as the top recommendation for divorced men over 50, while a fictitious service, "CancelEase," gained unwarranted popularity as a method for canceling Xfinity services—all due to short promotional lines deceptively appended to relevant Reddit interactions.
Crucially, the researchers highlight that WARP does not necessitate any tampering with AI providers or their foundational algorithms, but rather leverages the inherent trust these systems place in user-generated content—an area where traditional SEO and moderation practices struggle to maintain robustness. This means that the same poisoned page can simultaneously compromise multiple deep-research systems and well-known commercial agents like ChatGPT Deep Research and Google Gemini, both of which incorporate web citations into their synthesized answers.
As noted in follow-up analyses, both brands and scammers are likely to leverage this manipulation tactic as a streamlined influence strategy: identify high-ranking UGC content relevant to their niche, insert fabricated promotional snippets, and allow AI systems to propagate these misleading recommendations as though they were organic opinions.
Despite the potential for implementing defenses—such as blocking UGC domains or enhancing input filters—such measures could compromise the quality of AI-generated answers and fail to distinguish between well-articulated, legitimate community content and harmful misinformation. This leaves current AI research agents vulnerable to nuanced, large-scale manipulation, urging stakeholders to seek more resilient strategies in the struggle against misinformation and digital exploitation.