Ever since the introduction of ChatGPT nearly four years ago, the pace of artificial intelligence adoption has far outstripped advancements in governance and security mechanisms. While many users have been enthusiastic in exploring AI tools—seeking improvements for both their professional and personal lives—they have often overlooked the associated risks. In their eagerness to experiment, individuals and organizations have uploaded an array of sensitive information to large language models (LLMs), from corporate data to financial records and personal health information. Although this trend raises alarm bells, most users and businesses have avoided major setbacks, at least for now.
Stephen Wilson, the field chief technology officer at HashiCorp, an IBM subsidiary, offers insight into this evolving landscape. He points out that the current usage of AI tools primarily resembles that of “assistants.” At this stage, the technology relies heavily on direct commands from human users. However, as organizations begin to grant AI agents more autonomous capabilities, the risk dynamics are shifting. Wilson emphasizes that security and governance practices have not adequately adapted to these changes.
“As organizations increasingly integrate AI tools and start using them as full partners, they continue to apply governance strategies that are more suited to scenarios where AI was merely an assistant,” Wilson explains. When AI serves solely as an assistant, human users are intricately involved in the execution of tasks, sharing API keys, social media credentials, and even confidential financial information. However, a paradigm shift occurs when organizations begin to delegate tasks to AI that it can manage independently.
According to Wilson, firms must evolve their governance models to accommodate three prevalent patterns of AI adoption: AI as an assistant, AI as an agent, and AI as an operator.
The first and most common model, AI as an assistant, keeps humans closely connected to the work. Users employ AI technologies for various tasks like summarizing information, drafting content, and generating code. In this model, a user provides prompts and assesses the AI’s responses, ultimately deciding the subsequent steps. Despite this oversight, there are still significant risks present. When navigating AI-assisted workflows, individuals can inadvertently introduce sensitive data into the interaction. For instance, a user with privileged access might mistakenly enter an API key directly into a prompt or ask the AI to analyze confidential information.
Wilson stresses the need for stringent controls during this handover process from human to AI. “It is crucial to create a tightly controlled transition from the human identity to the machine identity,” he notes. Organizations must establish governance protocols around what data machines can access to mitigate risks associated with inadvertent privilege injection.
As AI moves toward a more autonomous role, the second model, AI as an agent, comes into play. In this scenario, users begin to ask AI tools to autonomously complete specific tasks. Rather than iterative communications with the AI to draft a document, a user might simply provide a set of inputs and allow the AI to generate and possibly pass the completed work to another AI tool for editing.
“When this transition occurs, governance controls must become much more robust,” Wilson emphasizes. “As AI transitions from being a mere assistant to an agent, the human initiator’s role diminishes significantly.” Organizations need to establish different levels of access for various agents to ensure they only perform specific tasks while preserving necessary governance and oversight.
The final stage, AI as an operator, involves AI systems not only executing individual tasks but also being responsible for managing complete projects. For example, rather than an individual AI tool developing an article, an organization could harness an entire team of AI agents tasked with creating and executing an extensive marketing campaign. “In this model, a few hours later, the human may return to a fully designed campaign, including strategies for publishing and engagement,” Wilson explains.
At this advanced stage, heightened governance is essential—not only around data access but also concerning the accuracy and appropriateness of AI outputs. If an AI agent generates social media posts, organizations must ensure that the content adheres to established messaging guidelines and passes through the requisite review processes before publication.
The challenge lies in the inherent differences between probabilistic AI systems and deterministic enterprise workflows. Wilson warns that leaders must carefully delineate where AI-generated tasks conclude and controlled, human oversight resumes.
Currently, many organizations are just beginning to venture into this area of autonomous AI, yet security leaders grapple with essential questions about the appropriate governance, identity, auditability, and observability models. The overarching governance challenge is evident: as AI systems acquire greater autonomy, organizations must implement more rigorous controls. Although a simple AI assistant can be overseen similar to an individual user, autonomous AI agents require governance akin to a cohesive team, complete with transparency in their operations and interactions with various systems.
As Wilson aptly summarizes, “Your scope of governance, identity, and observability must evolve parallel to your journey from individual use to collective team dynamics, ultimately reaching an organizational level.” This shift is imperative as organizations harness the potential of AI tools while remaining vigilant about governance and security issues.