Ineffective Metrics in Security Operations Centers: NCSC Raises Concerns
The National Cyber Security Centre (NCSC) has made a significant observation about the prevalent metrics employed to gauge the effectiveness of Security Operations Centers (SOCs). According to the NCSC, these metrics can often be misleading and, in some cases, actively detrimental to the performance of security operations teams. This warning was articulated by Dave Chismon, the NCSC’s Chief Technology Officer for architecture, in a detailed blog post.
Chismon noted that many organizations tend to select metrics that are easily quantifiable, particularly ones that can be understood by individuals who lack expertise in security operations. Metrics like the “number of tickets processed” or “time taken to close a ticket” might appear straightforward; however, these can lead teams to prioritize rapid ticket closure over thoughtful investigation. The danger here is twofold: not only may this practice drive cybersecurity teams to overlook significant threats, but it could also encourage the dismissive treatment of false positives.
Additionally, the emphasis on metrics such as the “number of detection rules” can result in a disincentive for quality over quantity. Analysts may feel compelled to write an overwhelming number of detection rules, which, instead of enhancing security, could inflate the volume of false positives and introduce ineffective guidelines. A disproportionate focus on the sheer volume of logs collected—without considering the qualitative value of those logs—adds another layer of inefficacy to security operations, particularly if this data fails to enhance detection capabilities.
To remedy these issues, the NCSC asserts that the most crucial metric to focus on is the ability of SOCs to detect and respond to attacks promptly. This concept, referred to as time to detect (TTD) and time to respond (TTR), stands as the benchmark for evaluating a SOC’s efficiency and effectiveness.
Strategies for Improvement
Chismon suggests implementing red and purple teaming exercises as a means to assess TTD and TTR within SOC environments. While TTD and TTR might be the only hard metrics that reflect the efficacy of a SOC, Chismon acknowledges that SOC managers often wish to track additional metrics to effectively monitor their operations on a week-to-week basis. However, he underscores a critical caveat: these supplementary metrics—while valuable for internal consumption—should not be shared externally or even internally with SOC analysts themselves to prevent them from driving misguided behaviors.
With a focus on minimizing TTD and TTR, analysts are encouraged to possess a comprehensive understanding of both the evolving threat landscape and the assets they are tasked with safeguarding. Mastery of the tools at their disposal is equally essential, as is access to relevant data that allows for the identification of unusual behavior and the time allocated for proactive threat hunting.
Chismon proposes various strategies to enhance threat detection:
-
Hypothesis-led Hunting: Analysts should formulate hypotheses about potential attacks based on their knowledge of threat actors and their techniques, subsequently searching for evidence within logs.
-
Maximal True Positives/Minimal False Positives: It is advisable for SOCs to maintain stringent thresholds for acceptable false positive rates when evaluating the suitability of detection rules.
-
Threat Awareness Metrics: Metrics that track an analyst’s familiarity with different threat actors—such as documentation completeness and training report engagement—can be beneficial.
-
Analyst Expertise Tracking: Monitoring the level of training and certifications attained by SOC analysts signifies their proficiency in the tools they utilize.
-
Organizational Engagement: Keeping tabs on SOC engagement with the broader organization allows for the identification and escalation of suspicious activities.
-
Analyst Job Satisfaction: High levels of job satisfaction among analysts are vital, particularly when they are engaged in meaningful tasks such as understanding attacker methodologies and collaborating across organizational lines.
- Log Coverage: Tracking the percentage of relevant assets reporting the appropriate logs aids in minimizing blind spots, thereby enhancing overall security posture.
Chismon culminated his insights with a stark warning regarding the potential pitfalls of relying on ineffective metrics. He stated, “The wrong metrics will render a SOC ineffective and make the work environment miserable, with analysts relegated to ‘ticket monkey’ roles, focusing relentlessly on clicking through false positives while being chastised for serious oversights.” For organizations worried about falling into the trap of ineffective metrics, he advocated for the engagement of a red or purple team from reputable vendors to provide a concrete assessment of their SOC’s true performance and vulnerabilities.
In light of these observations, the NCSC’s guidance serves as a pivotal reminder for security teams to refine their metric selections and implement strategies focused on enhancing both detection and response capabilities, thereby fortifying their overall security infrastructure.