LAS VEGAS — In a recent experiment conducted by a team of security researchers, large language models (LLMs) were put to the test to determine their ability to write and detect convincingly deceptive phishing emails. The results, presented at Black Hat USA 2023 by team member Fredrik Heiding, showed that while AI technology can produce effective phishing lures, they still fall short of the sophistication of manually designed emails.
The team, comprised of renowned experts including Bruce Schneier, Arun Vishwanath, and Jeremy Bernstein, tested four commercial LLMs in experimental phishing attacks on Harvard students. The LLMs used in the experiment were OpenAI’s ChatGPT, Google’s Bard, Anthropic’s Claude, and ChatLlama, an open-source chatbot.
Heiding, a research fellow at Harvard University, pointed out that the advent of LLMs has significantly lowered the bar for creating convincing phishing emails. “GPT changed this,” he said. “You don’t need to be a native English speaker; you don’t need to do much. You can enter a quick prompt with just a few data points.”
To gauge the effectiveness of the LLMs, the team carried out an experiment involving 112 students who received phishing emails offering Starbucks gift cards. While generative AI vendors have imposed safeguards and restrictions to hinder the creation of phishing emails, Heiding stated that LLMs can still be used to create simple marketing emails that can be repurposed for attacks. “The only difference between a phishing email and a marketing email is the intention,” he explained.
The researchers compared ChatGPT with a non-AI model called V-Triad, developed by Vishwanath specifically for constructing sophisticated phishing emails. In the initial test, V-Triad proved to be the most effective with an approximately 70% click rate. The combination of V-Triad and ChatGPT came in second at just under 50%. On the other hand, ChatGPT alone had a lower click rate of around 30%, and the control group email performed the worst at about 20%.
According to Heiding, the ChatGPT email’s lower performance may be attributed to its failure to mention Harvard in the text, despite the prompt including the university name. However, in a different version of the test, ChatGPT achieved a nearly 50% click rate, while the V-Triad/ChatGPT combination led with an almost 80% click rate. “That’s super exciting,” Heiding remarked. “Basically, it means already, we can create emails almost semi-automatically — a little bit manually, but almost fully automated — that are [as good as or better than] humans.”
Although the ChatGPT emails exhibited lower click rates compared to other models, Heiding emphasized that the experiment demonstrated the potential of untrained general-purpose LLMs in rapidly generating effective phishing attacks. He added that with LLM assistance, he expects phishing attacks to further improve.
In the second part of the experiment, the researchers assessed the LLMs’ capability to detect the intent of suspicious emails. ChatGPT, Bard, Claude, and ChatLlama were used to determine whether the emails were composed by humans or AI and to identify any suspicious elements. The LLMs were also asked to offer advice on how to respond to the emails.
Heiding revealed that the results were surprising and encouraging. While the LLMs excelled at classifying marketing emails, they struggled to identify the intent of the V-Triad and ChatGPT phishing emails. However, when it came to identifying suspicious content, all four LLMs performed extremely well in distinguishing marketing emails from suspicious ones. Some of the LLMs even demonstrated proficiency in differentiating between human and AI-generated emails.
One standout model in terms of detection and providing insightful advice was Claude. Not only did it achieve high results in the detection tests, but it also offered valuable guidance to users, such as suggesting they check the company website or forward suspicious emails to verify their legitimacy. Heiding deemed Claude’s advice as excellent, underscoring the rapid advancement of these language models and their potential.
Overall, Heiding concluded that out-of-the-box LLMs performed admirably in flagging suspicious emails, without any specific training on security data. He emphasized that this technology is readily accessible to everyone and possesses considerable power in enhancing cybersecurity.
As AI technology continues to evolve, researchers and security experts are constantly exploring its potential for both malicious activities and defense mechanisms. With the right precautions and measures in place, society can harness the power of AI to safeguard against cyber threats more effectively.