Thorough, independent testing is essential for organizations to assess the capabilities of cybersecurity vendors and protect themselves against increasingly sophisticated threats. One widely trusted evaluation is the annual MITRE Engenuity ATT&CK Evaluation, which analyzes vendor solutions based on their performance in defending against real-world attack scenarios.
Evaluating cybersecurity vendors based on their own claims is nearly impossible, which is why the MITRE results are so valuable. In addition to vendor reference checks and proof of value evaluations, the MITRE evaluation provides an objective assessment to help organizations make informed decisions about their cybersecurity strategy.
MITRE Engenuity performs the ATT&CK Evaluation by testing endpoint protection products against simulated attack sequences created by well-known advanced persistent threat (APT) groups. The 2023 evaluation focused on the attack sequences of Turla, a sophisticated threat group based in Russia that has targeted victims in over 45 countries.
Importantly, MITRE does not rank or score vendor results. Instead, it publishes the raw test data along with online comparison tools. This allows buyers to evaluate vendors based on their specific priorities and needs. The vendors’ interpretations of the results are subjective and should be taken into consideration.
Analyzing the results of the MITRE ATT&CK Evaluation can be challenging, as they are not presented in a format that many people are used to. Independent researchers often identify “winners” to simplify the process of selecting the top-performing vendors. However, in MITRE’s case, determining the “best” vendor is subjective, making it important to understand what to look for in the results.
The most important measurements to consider when reviewing the results are overall visibility and detection quality. These two factors indicate a solution’s ability to accurately and effectively detect threats. Threat visibility measures the number or fraction of detections out of a total of 143 possible chances in the Turla attack sequence. It is crucial to prioritize detections without configuration changes, as these reflect real-world scenarios.
Analytic detections, which provide information about why an activity is happening and how it is executed, are also important to consider. Vendors may not have provided analytic information for each step of the attack sequence, so it is best to prioritize those that do offer this information.
A chart illustrating the visibility and detection quality of each vendor can be a helpful tool for assessing their performance. Missed detections and poor-quality detections can lead to breaches or unnecessary work for security analysts.
To further understand the results of the MITRE ATT&CK Evaluation, cybersecurity leaders can attend webinars or review full analysis reports. These resources provide expert advice and insights into the evaluation and help organizations find the vendor that best fits their specific needs.
In conclusion, the MITRE Engenuity ATT&CK Evaluation plays a critical role in assessing the capabilities of cybersecurity vendors. By conducting thorough testing against real-world threats, it provides valuable insights for organizations in their decision-making process. Understanding the measurements used in the evaluation and prioritizing visibility and detection quality can help organizations make informed choices to protect against sophisticated attacks.