The weaponization of generative AI tools, specifically ChatGPT, is slowly but surely becoming a reality. Online communities are collaborating to crack ChatGPT’s ethics rules, known as “jailbreaking,” while hackers are busy developing new tools to harness or create large language models (LLMs) for malicious purposes.
Just like the popularity it has gained in the public eye, ChatGPT has also generated frenzy in underground forums. Since December, hackers have been actively searching for innovative prompts to manipulate ChatGPT and seeking open-source LLMs that can be repurposed for their nefarious pursuits.
According to SlashNext, a growing LLM hacking community has emerged, consisting of individuals armed with clever prompts but lacking AI-enabled malwares that truly make an impact.
Prompt engineering involves craftily manipulating chatbots like ChatGPT by asking them questions that trick them into breaking their programmed rules. Patrick Harr, CEO of SlashNext, describes this process as “looking around the guardrails” and continuously altering prompts to achieve specific outcomes.
Due to the repetitive nature of this task and the fact that everyone is targeting the same entity, online communities dedicated to jailbreaking have formed. Members support and assist each other in finding ways to exploit ChatGPT and make it perform tasks that its developers intended to prevent.
Though prompt engineering can only achieve limited results against a resilient chatbot like ChatGPT, the more concerning trend is the emergence of malware developers programming LLMs for malicious purposes.
In July, the introduction of WormGPT marked the beginning of the proliferation of malicious LLMs. This black-hat alternative to GPT models is specifically designed for activities such as business email compromise (BEC), malware, and phishing attacks. It is marketed as a tool without ethical boundaries or limitations, offering cybercriminals the ability to carry out such attacks at scale and with more precision.
Since WormGPT, similar products have surfaced in shady online communities, such as FraudGPT, advertised as a chatbot without limitations or rules. Additionally, DarkBART and DarkBERT, based on Google Bard, have made their appearance. These cybercriminal chatbots boast integration with Google Lens for image analysis and provide instant access to a vast cyber-underground knowledge base.
SlashNext reports that these malicious LLMs are proliferating rapidly, with many of them built upon open-source models like OpenAI’s OpenGPT. Less skilled hackers often customize these models, package them with a mysterious-sounding “___GPT” name (e.g. “BadGPT,” “DarkGPT”), and offer them for use, providing few limitations and ensuring user anonymity.
Despite the rise of WormGPT, its derivatives, and prompt engineering, SlashNext suggests that businesses are not yet under significant threat. However, the growth of underground jailbreaking markets indicates that cybercriminals have access to more tools, potentially leading to a shift in social engineering techniques and defense strategies.
Harr advises against relying solely on traditional training methods, as these attacks have become highly specific and targeted. He suggests that AI threats require AI protections, emphasizing the need for AI tools that can detect, predict, and block these evolving threats.
In conclusion, the weaponization of generative AI tools like ChatGPT has sparked the interest of hackers and online communities. While prompt engineering is commonly used to manipulate chatbots, the emergence of malicious LLMs presents a more significant concern. As the underground market for jailbreaking tools grows, businesses must stay vigilant and adopt AI-based defenses to counter the ever-evolving AI cyberweapons.