Seven months into Anthropic’s ambitious Project Glasswing, the initiative is facing growing criticism. The concept, which aimed to leverage cutting-edge AI models for identifying vulnerabilities in software, has left many maintainers feeling overwhelmed by the volume of findings, many of which appear to be underwhelming. This unfavorable assessment comes from Patrick Garrity, a vulnerability researcher at VulnCheck, who shared his insights in a report dated November 9.
Garrity’s recent findings are part of an ongoing examination of Project Glasswing’s efficacy, enabling scrutiny of whether Anthropic’s claims of success are genuinely substantiated or excessively exaggerated. The initiative, which represents Anthropic’s flagship endeavor, was designed to collaborate with security researchers, offering access to its advanced AI systems to combat software vulnerabilities more effectively.
According to Project Glasswing’s coordinated vulnerability disclosure dashboard, the project has reportedly uncovered an impressive total of 37,966 vulnerabilities. However, of these findings, only 7,349—approximately 19.4%—are formally recorded in Anthropic’s public vulnerability tracking register. Among these logged vulnerabilities, Garrity identified that 5,276 were validated as unique threats, while others included 1,654 duplicates and 419 cases of withdrawn reports.
The implications of these statistics are serious for those involved in maintaining software. Garrity pointed out that out of the validated vulnerabilities, approximately 13%—or 4,866—had been reported to the respective maintainers. Meanwhile, 193 cases remained in a “pre-disclosure” stage, meaning they had not yet been disclosed to the maintainers. Alarmingly, only 217 of these vulnerabilities, which amounts to less than 0.5% of the total claims made by Project Glasswing, have actually been resolved. This stark contrast raises significant questions about the project’s overall impact.
Building on his previous assessments made in September, Garrity observed that between then and November, Anthropic had added 4,613 findings to its ledger. Yet, within this timeframe, only 15 of these were marked as fixed, signaling a troubling trend in addressing security issues. A large chunk—1,668 of the recently added vulnerabilities—were categorized as duplicates, further complicating the situation.
This situation raises critical concerns about Anthropic’s claimed true positive rate of 94.2%. Garrity argues that this figure, while seemingly impressive, may not accurately account for the high volume of duplicates and withdrawn reports, suggesting that these metrics require reevaluation.
One contributing factor to the disparity between the number of vulnerabilities claimed and those that have been resolved may be the human limitations in processing and addressing vulnerability reports. In an update dated October 8, Anthropic acknowledged the challenge it faces, reporting that while over 29,000 candidate vulnerabilities were detected, the manual review had only covered approximately 6,000. This highlighted a “bottleneck” in their capacity to validate the findings effectively.
In a bid to streamline the process of vulnerability disclosure, Anthropic launched its own Open Source Security Scanner (OSS Scanner). This new self-service tool aims to emulate Google’s OSS-Fuzz project, facilitating maintainers in requesting scans and thereby managing vulnerability findings more independently. Furthermore, Anthropic disclosed that they had directly sent nearly 5,000 unverified reports to maintainers who opted for bulk reports. This adjustment has shed light on some of the challenges inherent in AI-generated vulnerability reports.
A pertinent example was shared by Daniel Stenberg, the maintainer of the well-known Client and URL (cURL) project. On November 6, Stenberg reported receiving 45 vulnerability notifications from Anthropic via Project Glasswing. However, after analyzing 34 of those reports using the OSS Scanner and his own tools, it became clear that only a fraction of the reports represented genuine vulnerabilities. Specifically, 19 were found to be non-issues, while six were classified as low severity.
Stenberg’s comments encapsulate a broader concern regarding the current state of Project Glasswing. He emphasized the potential of AI as a tool in vulnerability detection but also remarked that the reality of the project’s outcomes doesn’t align with the optimistic expectations set during its launch. With the majority of the claimed vulnerabilities neither being addressed nor exploited, and the burden of triage shifting back to open-source maintainers, concerns are intensifying.
Ultimately, Garrity concluded that while the intentions behind Anthropic’s Project Glasswing are laudable, the practical results are far from encouraging, with unresolved discrepancies persisting between expectations and realities. Infosecurity reached out to Anthropic for a comment on these developments, but the company had not provided a response by the time of publication. The ongoing conversation surrounding the success and transparency of Project Glasswing remains crucial as it unfolds.