The chatbot ChatGPT, which is powered by artificial intelligence (AI), has been found to be vulnerable to exploitation by attackers, according to new research from the Voyager18 team at Vulcan Cyber. The team found that by exploiting so-called “AI package hallucinations,” hackers can create malicious code packages that appear to be recommended by ChatGPT, and which developers may inadvertently download and incorporate into their software. This poses a significant risk to the software supply chain, as it can allow trojans and other malware to infiltrate legitimate applications.
ChatGPT, like other large language models (LLMs) used in generative AI platforms, generates responses to questions based on the data available to them online. While this can result in responses that are plausible, they can also be wildly inaccurate or false. LLMs like ChatGPT have been observed generating plausible but fictional information, extrapolating beyond their training, and producing responses that seem plausible but are not necessarily accurate. This phenomenon is known as an AI hallucination.
The researchers at Voyager18 discovered that AI hallucinations can be used by attackers to publish their own malicious versions of suggested code packages that don’t actually exist. The next time a user asks ChatGPT a similar question, they may receive a recommendation to use the now-existing malicious package, which can then be incorporated into a legitimate code repository or application. Lanyado warns that “a developer who asks a generative AI like ChatGPT for help with their code could wind up installing a malicious library because the AI thought it was real and an attacker made it real.”
While it can be difficult to spot malicious code that is effectively obfuscated or masquerading as a legitimate package, there are several ways that developers can validate the libraries they download. For example, they can check the creation date, the number of downloads and comments, or a lack of comments and stars, and look at any of the library’s attached notes. If anything looks suspicious, they can think twice before installing it.
This isn’t the first time that ChatGPT has presented a cybersecurity risk. In recent months, scammers have used it to steal user business credentials, and attackers have stolen Google Chrome cookies through malicious ChatGPT extensions. Phishing threat actors have also used ChatGPT as a lure for malicious websites.
Some experts believe that the security risk presented by ChatGPT is being overhyped, but many acknowledge that it exists due to the widespread adoption of generative AI platforms. Millions of people have embraced ChatGPT at work, including developers who have turned to the chatbot instead of online sources such as Stack Overflow. This creates a major opportunity for attackers to take advantage of the platform’s vulnerabilities.
As with any new technology that quickly attracts a solid user base, ChatGPT has also drawn bad actors aiming to exploit it for their own opportunity. This real-time example demonstrates that it’s important to remain vigilant and take measures to protect against cyber threats.