HomeMalware & ThreatsAre Small Models Narrowing the Gap with Frontier AI Cyber Tools?

Are Small Models Narrowing the Gap with Frontier AI Cyber Tools?

Published on

spot_img

Artificial Intelligence & Machine Learning,
Governance & Risk Management,
Next-Generation Technologies & Secure Development

Fable 5 Release Fuels Debate Over Whether Frontier Models Are Worth the Higher Cost

Are Small Models Narrowing the Gap with Frontier AI Cyber Tools?
Anthropic’s release of Fable 5 is fueling the debate over whether frontier AI is an essential tool for security, with smaller models increasingly matching frontier systems on routine security tasks. (Image: Shutterstock)

The ongoing discourse within the security industry regarding frontier artificial intelligence models has reached new heights following the release of Fable 5 by Anthropic. This model, classified as a Mythos-class system, is now publicly accessible and boasts cybersecurity capabilities channeled into a model that is less powerful. This executive move has propelled an essential question into the limelight: to what extent is access to frontier models necessary for effective security practices?

For several months, the security sector has categorized frontier AI as a distinct category, characterized by its vast capabilities that warranted restricted access. These frontier models, part of projects like Project Glasswing at Anthropic and OpenAI’s Daybreak, have been part of selectively approved programs made available solely to vetted partners. The rationale behind this stringent curation lies in the models’ superior aptitude for reasoning across extensive codebases, piecing together individual vulnerabilities to create functional exploits, and identifying obscure flaws that may have evaded detection for years.

Traditionally, smaller language models, designed to operate on limited data sets and fewer tokens, were deemed inadequate for such reasoning tasks. However, a shift in perspective is emerging as industry experts begin to explore whether these smaller models can bridge the gap and the conditions under which they might do so.

Identifying flaws within a specific section of code is a straightforward task for AI when it is directed appropriately. However, the challenge escalates when the need arises to perform reasoning across an entire codebase to uncover a vulnerability that has yet to be recognized. This complexity is where smaller models typically struggle.

Philippe Dourassou, the AI pen test lead at Aikido Security, explains the performance disparity with compelling statistics. He asserts that a less expensive model may succeed in complex identification steps only 30% of the time, compared to an 80% success rate for a frontier model. As the process continues deeper into a compromised system, these probabilities significantly fluctuate. While smaller models may complete the entire sequence of tasks only approximately 3% of the time, frontier models achieve this nearly half of the time. “The more complex and lengthy the task, the better the performance of the advanced model,” he notes.

Dipto Chakravarty, CTO at Black Duck, adds another layer to this conversation, emphasizing that the keys to success hinge on understanding how frequently these intricate tasks occur in practice. “In many cases, organizations do not require groundbreaking exploit chains; what they need most is reliable detection and triage conducted at scale,” he asserts. This perspective places significant importance on the surrounding infrastructure—the framework that determines which code gets scrutinized, how numerous testing attempts are allowed, and how findings are organized—as equally crucial as the models themselves.

When Anthropic conducts its Terminal-Bench tests, utilizing 1 million tokens per task with five attempts and an upper computing limit, a knowledgeable principal engineer might argue that a substantial portion of the observable capability difference stems from these controlled conditions rather than from the inherent intelligence of the model.

This perspective gains support from Microsoft’s findings during their experiment termed MDash, which paired frontier models for intricate reasoning with smaller distilled models—compressed versions developed for specific tasks—across a coordinated pipeline of over 100 agents. This approach yielded impressive results on CyberGym, a benchmark involving 1,507 real-world vulnerability reproduction tasks, where MDash outperformed both Mythos and GPT-5.5. Notably, no singular model was responsible for this performance; rather, the efficacy lay within the comprehensive pipeline.

For the high-volume, repetitive tasks that occupy the majority of security teams’ routines—including correlating findings with established vulnerability categories and aligning alerts across various systems—domain-trained small models have already demonstrated superior accuracy compared to frontier models. IBM Research’s CyberPal 2.0, a suite of security-specialized models, notably outperforms GPT-4o and o1 on key threat investigation activities.

George Gerchow, chief security officer at Bedrock Data, emphasizes that the specific results achieved by smaller models differ from those produced by frontier systems. He asserts, “The vulnerabilities in question have remained undetected despite multiple static analysis tools and pattern matchers being directed at the same code. They aren’t overlooked because of negligence; rather, their discovery requires nuanced reasoning that spans thousands of lines of context to identify connections that weren’t previously recognized.” He stresses that this is not merely a search issue but a reasoning challenge, firmly placing the reasoning capacity limitations at the forefront of model comparisons.

In most evaluation settings, smaller models are given access to the pertinent code directly, while an authentic autonomous scanning process must commence from an entire codebase to unearth the relevant sections.

The landscape of AI-generated vulnerability reports is shifting dramatically, as submissions are now flooding in faster than human reviewers can manage. Earlier this year, HackerOne even suspended its internet bug bounty program due to overwhelming AI-generated submissions, with the cURL project also experiencing similar challenges, necessitating the closure of its bug bounty program. Gerchow highlights that frontier models’ reasoning ability is irreplaceable, as they not only assess the existence of a flaw but also its reachability, exploitability, and whether it warrants action.

Howie Koh, Forescout’s vice president of innovation, summarizes the argument succinctly: “Both perspectives address different layers of the same issue. Smaller models excel in continuous, cost-effective scanning, while frontier AI is reserved for more in-depth analyses when the depth of reasoning becomes critical.”

This creates a scenario where multiple models can coexist within a single operational framework, potentially unveiling a market gap for vendors specializing in optimizing for outcomes and return on investment, rather than solely developing systems tailored to their proprietary frontier models. Vendors devoid of a specific proprietary model can select the most suitable tool for each task, rather than defaulting to utilizing the most premium solution available.

The balancing act encompasses a substantial consideration: “It only takes a single zero-day vulnerability identified by the frontier model that a smaller model overlooks to vindicate the investment in operating a frontier model,” Gerchow points out. With the introduction of Fable 5, security teams now possess an additional avenue to pursue, tailored to their specific challenges and budgetary constraints.

Source link

Latest articles

From Documents to Machine Intelligence

Policy as Code Revolutionizes Compliance and Governance Practices in Organizations In recent years, organizations have...

Novo Nordisk Reveals Breach of Clinical Trials Data

Novo Nordisk, a prominent Danish pharmaceutical company renowned for its role as a leading...

New Agentjacking Attack Compromises AI Coding Agents to Run Malicious Code

A newly uncovered class of attack, referred to as "Agentjacking," has emerged, capable of...

WhatsApp Uncovers Spearphishing Attempts Linked to NSO Group

WhatsApp Seeks Court Action Against NSO Group for Violating Injunction In a significant legal move,...

More like this

From Documents to Machine Intelligence

Policy as Code Revolutionizes Compliance and Governance Practices in Organizations In recent years, organizations have...

Novo Nordisk Reveals Breach of Clinical Trials Data

Novo Nordisk, a prominent Danish pharmaceutical company renowned for its role as a leading...

New Agentjacking Attack Compromises AI Coding Agents to Run Malicious Code

A newly uncovered class of attack, referred to as "Agentjacking," has emerged, capable of...