Google has issued a warning through its Threat Intelligence Group (GTIG) that cybercriminals and state-affiliated actors are increasingly utilizing AI for fraudulent activities, hacking, and propaganda campaigns. The report is based on a thorough analysis of how these threat actors are interacting with Google’s AI-powered assistant, Gemini. The study reveals that advanced persistent threat (APT) groups, cybercriminals, and information operations (IO) actors are leveraging AI to automate phishing scams, spread misinformation, and manipulate models to bypass security controls. While AI has not yet revolutionized cyberattack capabilities, threat actors are using it to enhance and scale their existing tactics.
The GTIG stated, “Rather than enabling disruptive change, generative AI allows threat actors to move faster and at higher volume.” Skilled actors are utilizing generative AI tools as a framework similar to the use of Metasploit or Cobalt Strike in cyber threat activity, while less skilled actors are using AI as a learning and productivity tool to develop tools more quickly and incorporate existing techniques.
The research conducted by GTIG has identified a surge in cybercriminals exploiting AI for activities such as business email compromise (BEC), phishing attacks, and the development of malware. Underground marketplaces are actively selling jailbroken AI models that can bypass security restrictions, enabling automated cybercrime.
Illicit AI tools like FraudGPT and WormGPT are being advertised in underground forums, offering capabilities such as automated phishing, AI-assisted malware creation, and cybersecurity evasion techniques. Cybercriminals are using AI to craft deceptive emails, manipulate digital content, and execute fraud schemes on a large scale.
State-backed APT groups are also exploring AI for cyber espionage and reconnaissance purposes. Iranian, Chinese, North Korean, and Russian APT actors have been observed attempting to use AI to analyze vulnerabilities, aid in malware scripting, and conduct reconnaissance on targets.
However, despite these efforts, GTIG found no evidence that AI has significantly enhanced the attack capabilities of these groups. APT actors mainly utilize AI for automating research, translating materials, and generating basic code rather than developing novel cyberattack techniques. Attempts to override AI safety mechanisms and generate explicitly malicious content have mostly been unsuccessful.
GTIG’s report also delves into how IO actors are leveraging AI for propaganda and misinformation purposes. Iranian and Chinese IO groups are using AI to refine messaging, generate politically motivated content, and enhance social media engagement strategies. Russian actors are exploring AI for automating content creation and extending the reach of disinformation campaigns.
Some groups are experimenting with AI-generated videos and synthetic images to create more compelling narratives. While AI has not yet revolutionized influence campaigns, threat actors are actively testing its potential to scale and refine disinformation tactics.
To combat the increasing misuse of AI, Google has strengthened its AI security measures under the Secure AI Framework (SAIF). The tech giant has expanded threat monitoring, adversarial testing, and real-time abuse detection to mitigate the risks associated with AI-powered threats.