A recent study conducted by Cobalt has revealed a growing skepticism regarding the efficacy of automated AI testing for identifying cybersecurity vulnerabilities. This erosion of confidence stems largely from a staggering number of false negatives that have emerged in the field. The findings were detailed in the Cobalt State of Pentesting Report 2026, which is based on comparative surveys conducted during 2025 and 2026 with approximately 450 cybersecurity professionals participating.
Notably, the report highlights a sharp decline in the percentage of organizations that depend solely on AI automation for testing vulnerabilities. This figure plummeted from 29% in 2025 to a mere 9% in 2026. In contrast, a significant shift has occurred towards a hybrid testing model, with nearly half (47%) of respondents expressing a preference for this approach. This model combines human expertise with AI capabilities to enhance vulnerability detection.
The findings are striking: over three-quarters (78%) of respondents indicated that fully automated scanning tools failed to detect critical vulnerabilities. Given the complexity of the cybersecurity landscape, this shortcoming is particularly alarming.
One of the most significant shifts observed was the increase in organizations adopting a hybrid model, which surged by 22 percentage points within just one year. Additionally, the utilization of automated tools specifically for low-risk environments rose by a comparable margin of 22 points to reach 47%. This suggests a trend towards more discerning use of automation, particularly in environments deemed less risky.
Andrew Obadiaru, CISO of Cobalt, commented on this trend, stating, “While the industry is rightfully excited about the potential of Mythos-class tools, unguided algorithms are inherently prone to returning even more false positives and costly false negatives than the automated scanners we have today.” His remarks underscore the importance of employing human oversight even as organizations explore the promise of advanced AI-driven tools.
The Expanding AI Attack Surface
A key factor contributing to the diminishing trust in automated AI testing is the intricate nature of the AI attack surface that these testing tools are expected to scrutinize. The report indicates that nearly one in three findings from an AI penetration test is rated as high risk, a figure that is 2.7 times higher than the average risk associated with traditional software.
At the time of the analysis, a concerning statistic was presented: less than two-fifths (38%) of vulnerabilities identified in large language models (LLMs) had been resolved. This leaves a staggering 62% still unaddressed, marking it as the lowest resolution rate across any asset class examined. Furthermore, the mean time to resolve (MTTR) for security issues pertaining to AI and LLMs saw a significant increase, rising from 19 days to 36 days during the reporting period. Cobalt posits that this trend indicates teams are facing increasingly challenging vulnerabilities, which require more time and resources to address.
Obadiaru elaborated, stating, “LLM vulnerabilities are deeply context-dependent and invisible to tools that lack an architectural understanding of the application.” He emphasized the need for a balanced approach to automation, suggesting that while automation should be deployed in areas where it excels, elite human expertise remains vital for uncovering and remediating complex business logic risks.
According to the report, organizations grappling with incidents related to AI identified shadow AI as the most common vector (44%), followed closely by concerns around data or model poisoning (41%) and improper output handling (41%). Other notable vectors included supply chain vulnerabilities (35%) and prompt injection attacks (34%).
Despite these alarming challenges, it is worth noting that while 60% of security professionals acknowledged the necessity for stronger LLM testing capabilities, only 42% indicated plans to bolster human-led red team operations. This disparity raises questions about the direction of security strategies in an increasingly complex threat landscape.
In summary, the findings of the Cobalt report serve as a clarion call for organizations to reevaluate their approach to cybersecurity testing. The trend towards hybrid models highlights a recognition of the limitations of automated testing and underscores the enduring importance of human expertise in the fight against cyber threats. As the landscape evolves, the partnership between human insight and AI-driven tools will be essential to effectively navigate new vulnerabilities.