Shift from Full AI Automation in Penetration Testing: A Return to Human Expertise
In a significant turnaround, security teams are increasingly moving away from relying solely on automated AI penetration testing, following a year marked by disappointing outcomes. According to the recently published Cobalt’s AI and Pentesting Pulse Report 2026, which surveyed 455 professionals in the cybersecurity field, the percentage of organizations depending exclusively on AI automation for security testing has plummeted from 29% in 2025 to a mere 9% this year. This drastic decline is largely attributed to the realization that automated AI scanning tools frequently miss critical vulnerabilities and generate false negatives, as reported by 78% of organizations surveyed. Consequently, 47% of respondents are now adopting hybrid models that incorporate both AI testing and human expertise, combining the strengths of technology with the nuanced understanding that human testers provide.
The limitations inherent to AI-only security testing stem from fundamental gaps in how automated tools comprehend applications. Gunter Ollmann, the Chief Technology Officer (CTO) at Cobalt, elaborates on the challenges faced by these automated systems. He points out that seasoned penetration testers have a deep understanding of business logic, user intent, application context, chained exploitation paths, and subtle trust relationships—elements that current AI systems often overlook. Meanwhile, security engineer Noelle Murata emphasizes that while AI can list potential attack surfaces more quickly than human analysts, it falls short when it comes to adaptability. Instead of employing creative strategies to exploit vulnerabilities, AI tends to brute-force every possible permutation, which not only diminishes effectiveness but also drives up costs compared to skilled human testers who can effectively target their efforts.
Particularly problematic are applications that utilize AI or large language models (LLMs), which present unique security challenges. These systems generate high-risk findings at nearly three times the rate of conventional software; alarmingly, only 32% of those findings ever get addressed, marking the lowest resolution rate tracked by Cobalt. This issue is intensified by the complex nature of AI vulnerabilities, which cannot be remedied through simple code patches. Instead, they often necessitate modifications to prompts, adjustments in model behavior, improvements in data governance, retrieval systems, agent permissions, or even comprehensive changes to the architecture of the application itself. Attack vectors unique to AI, such as prompt injection and insecure model integrations, further complicate matters, revealing areas where security teams generally lack the training and experience compared to traditional vulnerabilities like SQL injection.
The fundamental architecture of AI systems stands in stark contrast to nearly two decades of established application security practices. Traditional application security (AppSec) principles emphasize constraining inputs through parameterized queries and validation processes designed to reject unexpected structures. Conversely, AI and LLMs are designed to interpret and act upon ambiguous natural language prompts, connecting to internal knowledge bases, customer data, and privileged APIs. Denis Calderone of Suzu Labs adds that AI systems operate on a probabilistic rather than deterministic basis. This unpredictability means a prompt injection attack might only be successful three out of ten times, depending on the context or conversation history, making testing and remediation exponentially more challenging than for traditional software.
In light of these revelations, security experts recommend that organizations leverage AI for preliminary functions like reconnaissance, broader coverage, and repetitive validation tasks. However, they stress that the more nuanced judgments concerning business logic, chained exploits, and innovative attack paths should be left to human testers. Additionally, experts advocate for conducting binary-level analysis of compiled code, containers, and dependencies to identify issues like tampering, embedded secrets, and supply chain risks that may escape detection through logic-level penetration testing. By testing all components prior to deployment, organizations can gain a comprehensive overview of what enters production environments, allowing them to catch malicious code or tampering before systems go live.
In conclusion, the industry appears to be undergoing a pivotal transformation, moving away from an over-reliance on automated AI systems toward a more balanced approach that integrates human expertise. This shift reflects a growing recognition of the limitations of current AI technologies in the complex landscape of cybersecurity threats, emphasizing the need for a hybrid model that maximizes both technological efficiency and human ingenuity. As organizations adapt to this evolving environment, they can expect to enhance their vulnerability detection and resolution strategies significantly.