Federal Advisory Outlines Five Stages for Detecting and Containing AI Incidents
U.S. security agencies have recently unveiled a comprehensive guide aimed at assisting the evolving artificial intelligence sector in effectively preventing, preparing for, and responding to incidents involving rogue AI agents. The ground-breaking document was jointly released by the National Security Agency (NSA), the National Institute of Standards and Technology (NIST), and the Cybersecurity and Infrastructure Security Agency (CISA).
The advisory, disseminated on Friday, seeks to establish voluntary practices for managing the intricacies associated with AI training, evaluation, and deployment. The guidance is particularly relevant in light of growing concerns regarding agentic systems that might lack adequate safeguards and could undertake unintended actions or act beyond their authorization. This alarm has been amplified by revelations from prominent AI firms such as OpenAI, Anthropic, Google, and Meta, all of which have disclosed instances of their models behaving unpredictably during tests and simulations.
Experts in the field have acknowledged the significance of this guidance. Chris Hughes, the vice president at Noma Security, a platform focused on agent security and governance, emphasized its timeliness. He pointed out that while implementing practices such as least-privilege access for agents, monitored egress, and establishing “kill switches” could drastically mitigate risks, they also need to be complemented by thorough monitoring across various layers including the model, host, and network. Hughes stressed the importance of validating any containment measures prior to a system’s restart, citing hard-earned lessons from previous incidents.
The guidelines articulate a structured approach encapsulated in five distinct stages of incident management: preparation, detection, response, recovery, and reporting. This framework builds on existing models such as NIST CSF 2.0 and SP 800-61r3, offering a familiar terminology and structure that resonates with security leaders while specifically addressing the unique challenges posed by highly capable AI agents.
Aviv Nahum, co-founder and CEO of Above Security, highlighted a fundamental challenge in this landscape: determining when an autonomous AI agent’s actions constitute a security incident. Nahum noted that not every unintended behavior of an AI agent is necessarily harmful; agents can perform unexpected actions while still operating within their permissions. Continuous monitoring is essential, enabling security teams to maintain awareness of what an agent aims to achieve, its interactions with other systems, and whether its behavior remains rational.
The government’s guidelines underscore the necessity of defining clear indicators for detecting rogue AI behaviors. It advocates the collection of telemetry data and the manual review of anomalies to bolster security efforts. Additionally, the advisory draws attention to the significance of “near-misses,” which are incidents where the agent’s actions, while not leading to immediate harm, carry foreseeable risks. Such near-misses should be treated with the same seriousness as actual incidents, necessitating immediate action to block, contain, or terminate the rogue agent.
Moreover, the advisory also points out that AI-powered monitoring tools could introduce additional vulnerabilities. Hughes advises that gateways and monitoring systems should be subject to rigorous scrutiny, suggesting the use of diverse models from different families for effective cross-validation.
The focus of this guidance rests primarily on autonomous AI models equipped with software tools that can interact with various environments. Notably, it does not address issues stemming from uninstructed agent activity resulting from adversarial attacks, such as prompt injections or malware. Despite recognizing these scenarios as critical areas for future work, Hughes criticizes the scope of the guidance for failing to include adversarial attacks in its definitions, suggesting such attacks are one of the most prevalent ways agents malfunction within enterprises.
Moreover, it is crucial to recognize that the recommendations primarily cater to frontier labs, applying minimal guidance to actual users. As Hughes stated, the next logical step is to extend this rigor to organizations that deploy AI agents on a daily basis. Ensuring that the guidelines bridge the gap between genesis labs and practical applications is essential for creating a comprehensive and robust infrastructure for the future of AI safety.