Generative models equipped with artificial intelligence (AI) features are set to be integrated into the productivity tools people use daily, such as word processors, email clients, and artistic software. They will even make their way into search engines and other apps, heralding a new generation of functionality. However, as with every new tool, there are potential downsides to the integration of generative AI into our daily lives. For instance, both benevolent and malevolent content could be created with it. The ability to determine whether content has been created by an AI alone will not be sufficient to determine its legitimacy. Evildoers will also have access to these technologies and be able to wield them in increasingly productive ways.
One major concern is how adversaries might use generative AI to create content designed to socially engineer people. Such content is almost always made to appear neither benign nor malicious. ChatGPT, for instance, could be told to write an email on behalf of someone in a hurry for feedback on a presentation, which might seem harmless enough. The software could also be used to send an email politely informing a person that they have accidentally knocked into someone else’s car in a company parking lot. It could even be used to retrieve confidential documents and upload them to repositories created in response to the General Data Protection Regulation.
Earlier this year, a study was conducted using a large language model to generate multiple types of undesirable content including phishing emails, fake news, and online harassment. The results indicated that AI models would not refuse to create social engineering content and would do a great job of producing the content requested. While access to systems like GPT-3 is still relatively expensive, with the development of more affordable and accessible models, AI-generated content is becoming a viable option for adversaries looking to attack victims.
One of the reasons why adversaries would prefer to use a language model to create phishing and spear phishing content is that these models are capable of producing various forms of English with a much higher accuracy rate than a human operator could manage up to this point. Additionally, these models can write in other languages, making them an appealing option for threat actors looking to create multilingual content. In the case of spear phishing, the software could be used to impersonate someone convincingly and build trust with the victim over many interactions.
Currently, there are no technological solutions to help individuals or companies definitively identify whether they have been socially engineered, and relying on vigilance and awareness is still the best approach. Media literacy and phishing awareness training can contribute to building vigilance within a company, and being taught about different psychological concepts within social engineering attacks could be beneficial as well. Companies could also reward employees who follow safety protocols and report threats.
As AI continues to permeate various functional areas, the emergence of new tools designed to combat these malicious attacks will themselves become a pressing concern. Until such a time, remaining vigilant and adding necessary precautions will be critical to ensuring that these attacks do not subvert companies at the enterprise level or exploit individuals.