Divergent Findings Among Leading AI Models in Vulnerability Audits
A recent study by software security firm Echo highlights significant discrepancies in vulnerability detection capabilities among three leading AI models when they were tasked with auditing the same open-source code base. The research illustrates that despite each model analyzing the same code—a set of two Maven libraries and one npm package—they produced markedly different findings, resulting in a mere one overlapping vulnerability out of 121 distinct potential issues identified.
Published on November 18, the study scrutinized how Anthropic’s Claude Mythos 5.1, OpenAI’s GPT-5.6 Sol, and Z.ai’s GLM-5.3 performed using Echo’s proprietary hunting framework and validation process. Notably, Echo’s team accessed GPT-5.6 Sol through Daybreak, a program developed by OpenAI to facilitate vetted security teams in their efforts.
The results were stark: a staggering 83% of the identified issues—amounting to 100—were flagged by a single model. The remaining two models only accounted for 20 of the issues. These findings raise crucial questions about the consistency and reliability of AI-driven vulnerability detection, particularly as the industry increasingly leans on automated systems for security assessments.
Discrepancies in Validation and Severity Ratings
Among the three models, Claude Mythos led in the number of validation passes, tallying 25, while GLM-5.3 and GPT-5.6 Sol reported nine and eight passes, respectively. However, the distribution of the vulnerabilities discovered revealed a different picture. GPT-5.6 Sol achieved six passes on a specific Maven library, while GLM-5.3 unearthed six validated issues that neither of the other models captured.
The severity ratings assigned to these vulnerabilities further highlighted the models’ divergent evaluations. For example, one flaw in a popular npm package, downloaded over 350 million times monthly, was designated as "Critical" by GLM-5.3. In contrast, Claude Mythos rated it as "High," and GPT-5.6 Sol dismissed it as documented behavior rather than an actual bug. Automated validation for this issue categorized it as "Low," illustrating the potential for significant interpretative differences among the models.
In terms of operational efficiency, the models exhibited considerable variance as well. GPT-5.6 Sol completed its auditing tasks in approximately 24 minutes. In contrast, Claude Mythos took nearly 198 minutes, while GLM-5.3 required around 174 minutes. Economically, GLM-5.3 emerged as the most cost-effective option, with its three audits costing just $22 compared to Claude Mythos’s $303 and GPT-5.6 Sol’s $30. Echo noted that these figures were reflective of its own access agreements.
Outcome Validity and Implications
The study initially generated 446 findings, which were later refined to 152 after data cleanup. Among these, 74 findings were categorized as High or Critical, and 55 went through automated proof-of-concept testing. Ultimately, 42 findings were validated. Nonetheless, Echo cautioned against assuming that these validated findings were all distinct vulnerabilities. An issue flagged by more than one model could be counted multiple times, complicating the analysis. Moreover, vulnerabilities rated as Medium or Low were not tested, resulting in known flaws, like one identified by Claude Mythos, failing to receive validation.
Yuval Saggie, a security researcher at Echo, emphasized that the quest to identify the "best model" for vulnerability discovery is misguided. He pointed out that if an attacker leverages one AI model while a defender relies on another, vulnerabilities might remain undetected altogether. Such findings underscore the urgency for organizations to consider the diversity of AI tools in their cybersecurity strategies.
The study served as a controlled comparison rather than a comprehensive benchmark. Echo clarified that the automated validations should not be regarded as confirmations from project maintainers. This distinction is vital, as it highlights the limitations of current AI vulnerability detection methods and the need for further evaluation.
In conclusion, the discrepancies among the AI models illuminate the complexities and challenges inherent in relying solely on automated systems for vulnerability assessments. Until the maintainers of the projects involved are notified, Echo has opted to withhold detailed technical information, ensuring responsible disclosure and prioritizing security in the audit process. The lessons drawn from this research will undoubtedly influence the future landscape of AI-driven security assessments, shaping how companies approach vulnerability management in an increasingly digital world.
