As artificial intelligence (AI) increasingly becomes integral to security operations centers (SOCs), automating critical tasks such as threat triage, indicator extraction, and incident report generation, a pertinent question has emerged within the cybersecurity community: Does the performance of a SOC depend more on the large language model (LLM) being utilized or on the quality of the underlying security data? This question is gaining traction as organizations strive to optimize their cybersecurity defenses.
Academic research presents a spectrum of views on this topic. A study from the journal Frontiers in Artificial Intelligence highlights varying degrees of accuracy, relevance, and clarity across leading AI models, including Claude 3.5 Sonnet, Gemini, ChatGPT 4o, and Mistral Large 2. Conversely, findings published in the Information Systems journal contend that "trustworthy AI applications necessitate high-quality training and test data across multiple quality dimensions, such as accuracy and completeness." This dichotomy underscores the need for further exploration into the true determinants of effective AI performance in the realm of cybersecurity.
Recent investigations lend credence to the argument emphasizing the critical importance of data quality. Research conducted under the umbrella of the "Provably Better Data" initiative reveals that while the choice of model plays a role, the fidelity of the data utilized can significantly dictate the success of security operations. These studies suggest that high-quality network evidence can amplify security outcomes by as much as 2 to 4 times across fundamental investigation metrics.
For Chief Information Security Officers (CISOs), these findings offer tangible operational advantages. High-fidelity data can:
-
Enhance Telemetry: By supplying accurate telemetry, organizations can reduce their mean time to respond (MTTR).
-
Manage Costs: Controlling token expenditure is crucial for budget considerations.
- Maximize ROI: Higher returns on security investments can be achieved, ultimately illustrating the efficacy of security teams to stakeholders.
The implications of these findings extend beyond mere academic contemplation; security practitioners may utilize this evidence to guide architectural decisions for future SOC strategies. Additionally, CISOs can leverage this research to substantiate investments in infrastructure, alleviate analyst burnout stemming from alert fatigue, and furnish robust security metrics to both executive teams and board members.
To properly assess the true factors influencing AI performance within enterprise security frameworks, the "Provably Better Data" project established a controlled experimental framework. The initiative aimed to evaluate model performance across two operational benchmarks: a Capture the Flag (CTF) scenario involving a 44-question investigation based on a Volt Typhoon attack and an incident response analysis derived from a Salt Typhoon dataset.
To ensure data quality was isolated as the sole variable influencing outcomes, the experiment utilized four distinct network telemetry sources under uniform conditions:
- Enriched logs from Corelight
- Open-source nDPI firewall logs
- Snort 3 intrusion detection system alerts
- NetFlow connection telemetry
Multiple iterations of each dataset were executed using an Open Cybersecurity Schema Framework (OCSF) normalized schema, with various models—including Anthropic Claude Opus 4.6 and Google Gemini Pro 3.1 Preview—tested against identical prompts. Model performance was quantitatively assessed based on CTF accuracy and incident response claims corroborated by available evidence.
The findings yielded compelling insights: although advanced AI models exhibit sophisticated reasoning capabilities, their conclusions are intrinsically limited by the evidence at hand. In situations where telemetry lacks crucial protocol-level context, AI agents are unable to speculate intelligently—essentially exposing the depth of their dependency on the quality of incoming data.
One notable discrepancy illuminated in the research involved the NetBIOS computer name for a specified IP address, which revealed stark differences in outcomes based on the source of the logs. Corelight logs furnished the precise answer due to their comprehensive details, while the firewall logs, despite indicating NTLM activity, failed to provide the necessary specificity to yield the correct response. This gap underscores how crucial operational insights can be hampered by incomplete data.
The "Provably Better Data" project demonstrated that enhanced data fidelity could translate into a 2 to 4 times better performance in security operations compared to standard logging practices. Accuracy rates observed during the CTF benchmark varied wildly depending on data sources:
- Corelight logs exhibited a stunning 95.2% accuracy.
- Firewall logs managed a mere 58.3% accuracy.
- Snort 3 alerts fell to 39.4%.
- NetFlow records languished at just 25.8%.
The outcome was unequivocal: richer telemetry significantly bolstered model performance, with Corelight’s data enabling a high CTF score of 4,178.3 points compared to NetFlow’s 970.0 points—a stark more than fourfold disparity.
The incident response evaluation corroborated these findings, illustrating that Corelight logs yielded a coverage rate of 90.3%. In stark contrast, the remaining sources paralleled the prior accuracy results, highlighting the pervasive impact of data quality on security investigations.
Moreover, as the research team delved deeper, they observed that not only did superior data lead to significant advances in capability, but the overall speed of investigation also improved dramatically. The model completed investigations in a mere 14.7 minutes when provided with high-quality Corelight logs, while lower-quality data sources extended the duration substantially.
The overarching takeaway from the research is clear: high-quality data significantly enhances cybersecurity efforts, revealing critical vulnerabilities that may remain obscured when relying on poorer quality logs. Furthermore, organizations must recognize that fundamental data deficiencies cannot be overshadowed by model upgrades or complex engineering alone.
As AI automation continues to revolutionize threat detection, indicator tracking, and incident management, security leaders are urged to prioritize the enhancement of evidence quality in conjunction with model selection. Future SOC investments should be reviewed with a discerning eye towards the implications of data quality on overall effectiveness.
For those interested in an in-depth analysis of these methodologies and findings, the detailed white paper regarding the "Provably Better Data" research is available on Corelight’s website. By understanding the significance of high-fidelity network evidence, organizations can enhance their security postures and respond more effectively to the ever-evolving landscape of threats.