Missing: Threat Models to Defend Against Attacks in the Age of Agentic AI
At the 35th annual Cryptographers’ Panel held during the RSAC Conference in San Francisco, experts expressed grave concerns regarding the rapidly evolving landscape of cybersecurity, particularly in light of advancements in artificial intelligence (AI). Panelists emphasized the need for a thorough understanding of how these developments affect current defenses against cyber threats.
The discussion was sparked by a notable assertion from Dawn Song, a professor at the University of California, Berkeley, who co-directs the Center for Responsible Decentralized Intelligence. She pointed out that AI agents have become remarkably proficient at identifying zero-day vulnerabilities in extensive, open-source software solutions. In an era where AI plays an increasingly pivotal role, it is predicted that AI-enabled code development tools could generate up to 90% of all new code within this year alone.
This unprecedented rise in AI agents represents a significant paradigm shift for cybersecurity professionals. The expanding capabilities of these tools raise pressing questions concerning how best to defend against enhancements in AI-enabled attacks. Participants at the panel highlighted various strategies, including the application of differential privacy methodologies to secure AI deployment, the integration of cryptography within deep neural networks, and the resolution of key management challenges related to quantum computing.
Among the most alarming aspects discussed during the session was the proliferation of AI tools, which particular panelist Adi Shamir, a co-founder of the RSA cryptosystem, described as "explosive." Many of these applications require extensive access to private information, including personal files and calendars. Anecdotal evidence illustrates how these AI tools can misbehave, leading to consequences such as the deletion of cherished family photos or the unintentional exposure of sensitive APIs.
Shamir articulated a cautionary viewpoint regarding AI agents, characterizing them as "very clever idiots." This characterization underscores the risks associated with their potential misuse. As reliance on these tools grows, cryptographers are left pondering whether current cryptographic systems can withstand the challenges posed by intelligent adversarial algorithms. The fear that AI might unveil new vulnerabilities harbors potential ramifications for the very foundations of cryptography, prompting essential inquiries about the adequacy of established systems.
An essential question posed by the panel’s moderator, Paul Kocher, a coauthor of the SSL/TLS protocol, was whether AI itself could represent a threat to cryptography. He remarked that cryptography is traditionally based on the premise of security emerging from complex mathematical problems. However, with the incorporation of AI technologies, the distinction between what is known and unknown becomes increasingly ambiguous. The panelists acknowledged that, to date, no AI tools have successfully identified new cryptographic vulnerabilities, primarily reiterating findings documented in existing literature.
Cynthia Dwork, a prominent computer science professor at Harvard University and distinguished inventor of differential privacy, suggested a proactive approach to addressing these concerns. She urged cryptography researchers to share their findings with AI teams striving to crack cryptographic systems, even before public disclosure. By providing early insights, researchers could facilitate the testing of AI’s capabilities to discover previously unknown weaknesses, offering a chance for better preparedness against future threats.
Panelists also drew parallels to the notable advances made through AI competitions in addressing complex problems, such as the protein folding challenge, which resulted in significant scientific breakthroughs. Yet, Dwork lamented that while AI has not yet managed to expose flaws in existing cryptographic systems, numerous cybersecurity implications are already emerging. She emphasized that the current understanding of the threat landscape remains insufficient; there is a pressing need to define new threat models that can better inform defense strategies.
One alarming prospect outlined by the panelists is AI’s capacity to swiftly synthesize information from diverse sources, enhancing its ability to personalize attacks. Dwork highlighted the potential for AI systems to leverage personal data for malicious activities, including blackmail, while simultaneously enabling unprecedented scales of traffic analysis with alarming surveillance implications. This accelerated maleficence, alongside the inherent shortcomings in existing defenses, raises significant concerns.
As the panel discussions evolved, Shamir accentuated the speed with which AI can transform how attackers operate. For example, he noted that spear-phishing attacks could soon occur at an alarming scale, facilitating exploitation of vulnerabilities within minutes after their announcement—well before patches can be deployed to mitigate these risks. Whitfield Diffie, known for the groundbreaking Diffie-Hellman key exchange, echoed this sentiment, stressing that the prolonged duration required to implement system patches represents a fundamental threat.
Looking to the future, the consensus among panelists was that AI currently tips the balance in favor of attackers. However, Diffie’s perspective varied slightly; he posited that the advantages of AI for attackers were equally available for defenders, provided they are willing to adopt proactive measures for fortifying their systems.
As the implications of AI on cybersecurity persist to unfold, Shamir concluded on a cautiously optimistic note. While he acknowledged the prevalence of attacking agents, he also expressed belief in the existence of defending agents that could be refined to combat these threats effectively—suggesting that the landscape could evolve into one where both human and AI entities coexist in a dynamic struggle for cybersecurity prowess.