OT Operators Shouldn’t Wait for Mythos Access to Probe Codebases
The recent government-ordered shutdown of access to Mythos 5, Anthropic’s most advanced large language model (LLM) known for its cyber capabilities, has significantly highlighted an important message from security experts aimed at the operational technology (OT) community: reliance on Mythos isn’t essential.
Patrick Miller, CEO of Ampyx Cyber, an OT security consulting firm, emphasized the urgency of this message, saying, “No, you don’t have to wait to do this.” Miller insists that various commercially available LLMs have the capability to detect software vulnerabilities effectively. Although the latest version of Mythos may outperform others, Miller points out that many LLMs can provide adequate vulnerability assessments.
The effectiveness of the vulnerability scanning process relies not only on the sophistication of the LLM but also significantly on the supporting software infrastructure known as a harness. This includes bespoke training focused on the specific codebase being evaluated and other contextual elements. "Train them on your code, train them on your development life cycle, and just go to town," he advises. Many companies in the OT sector have yet to access Mythos, and waiting for a model exclusively available to a select few — through Anthropic’s club called Glasswing — is not a strategic path forward.
Recent actions taken by the Trump administration have further complicated the situation. Using export control authority, the government mandated that access to both Mythos 5 and its publicly accessible version, Fable 5, be restricted to U.S. citizens. This prohibition meant that even foreign employees of Anthropic working in the United States would be barred from utilizing these models, forcing the company into a position where it had to consider how to verify users’ citizenship. Ultimately, Anthropic opted to pull these models from service entirely.
This drastic measure followed an incident where a researcher at Amazon successfully jailbroken Fable 5. Despite being equipped with protective features designed to prevent misuse, such as redirecting certain queries to less capable models, hackers found ways to bypass these barriers, raising significant security concerns.
Although Anthropic declined to comment on the implications of this decision or its potential impact on Glasswing, it appears likely that members of this exclusive group would revert to utilizing Mythos 4, a previous iteration of the model.
Miller’s concerns about companies delaying their vulnerability assessments resonate with broader sentiments within the OT security field. Moreno Carullo, CTO of Nozomi Networks, a dedicated OT security firm, echoed the sentiment, stating that “waiting for this model, or any other, is not the way to go." He urged companies to proactively stress-test their software against existing models, indicating that there remains a wealth of vulnerabilities to uncover.
The initial purpose behind Mythos was to provide companies a head start in identifying software vulnerabilities, enabling them to address potential weaknesses before adversarial actors could leverage more advanced models. Carullo pointed out that organizations which have not yet conducted thorough stress tests on their codebases are already at a disadvantage.
Research indicates that the capability of AI in discovering vulnerabilities can fluctuate considerably. Aisle, a company that specializes in AI-driven vulnerability assessments, found that LLM performance varies greatly and is influenced not only by their size and sophistication but also by multiple other factors. Their tests indicated that smaller, cheaper, open-source models could uncover vulnerabilities previously identified by the more advanced Mythos, emphasizing that success in vulnerability discovery is not solely tied to a model’s complexity.
However, experts caution against a narrow focus that fixates exclusively on vulnerability discovery through Mythos. Victor Wieczorek, senior vice president of offensive security at GuidePoint Security, suggests that organizations should leverage LLMs to alleviate mundane tasks beyond vulnerability assessments. He noted the potential for LLMs to streamline various pains within an OT context, thereby enhancing operational efficiency.
Despite their advantages, experts also acknowledge the limitations of LLMs. While these models excel in identifying vulnerabilities, they lack equivalent capabilities in remediating these issues. Matthew Butkovic, technical director for cyber risk and resilience at the Software Engineering Institute at Carnegie Mellon University, raised alarms about the developing barriers in sharing vulnerability information in an environment where AI technologies excel at identifying, but not fixing, issues.
He stressed the need for a balanced approach that considers the ramifications of rapidly deploying advanced capabilities, particularly in an arena where the potential for misuse and abuse exists. The day after the shelter-in-place order regarding Mythos was issued, senior technical personnel from Anthropic reportedly traveled to Washington to negotiate with governmental bodies, including the White House, CIA, and Department of Commerce, hoping to address the dispute and find a viable resolution.
In conclusion, as the landscape of AI and cybersecurity continues to evolve, organizations in the OT sector are urged to take proactive measures in testing and securing their software. With the resources available, they should not await access to specialized models like Mythos, but rather utilize existing tools to enhance their vulnerability detection and overall security posture.