The recently released results of the 2023 MITRE ATT&CK Enterprise Evaluation have caused a frenzy among participating vendors. As they strive to present themselves in the best light possible, some vendors are resorting to misleading tactics to make it appear as if they performed well, when the reality is quite different. This has become a common practice in the vendor community, making it challenging for those who are not familiar with the MITRE ATT&CK Evaluation process to accurately assess their performance. In this article, we will delve into the main evaluation categories, how they are measured, and what to look for when evaluating vendor performance. By understanding these key points, readers will be able to visit the participants’ websites and discern the truth behind their claims.
The MITRE ATT&CK Framework, developed by MITRE Engenuity, consists of 14 tactics that attackers employ to achieve their objectives. Real-life attacks can encompass any number of these tactics. Each tactic comprises multiple techniques that describe the actual activities carried out by adversaries to accomplish their goals.
The MITRE ATT&CK Enterprise Evaluation is an annual event where MITRE simulates an attack by replicating the tactics and techniques used by a known threat actor. The simulated attack sequence consists of multiple steps, which generally correspond to the tactics in the MITRE ATT&CK Framework. This year’s evaluation consisted of 19 steps, with some tactics repeated multiple times. For instance, “Lateral Movement” was used six times out of the 19 steps. An example of the steps used in the evaluation’s day one testing is as follows:
1. Initial Compromise
2. Initial Access
3. Discovery & Privilege Escalation
4. Persistence
5. Lateral Move to Domain Controller
6. Credential Access Discovery
7. Credential Access
8. Lateral Move to Linux
9. Watering Hole
The MITRE ATT&CK Evaluation breaks down into steps and sub-steps. Each step typically mirrors a tactic in the MITRE ATT&CK Framework, while sub-steps emulate techniques within those tactics. In the 2023 Evaluation, MITRE used 19 steps that were further divided into 143 sub-steps. The evaluation takes place over four days, focusing on different aspects each day:
Day 1: Evaluating the ability to detect and classify threats for scenario one.
Day 2: Evaluating the ability to detect and classify threats for scenario two.
Day 3: Reviewing any missed detections from the first two days and testing with configuration changes.
Day 4: Evaluating the effectiveness of the protection measures.
When discussing the MITRE ATT&CK Evaluation results, two terms often cause confusion: detection and visibility. It is crucial to understand these terms to accurately analyze the results. Detection refers to the number of steps detected, with a step considered detected if one or more sub-steps within it are detected. In other words, as long as a vendor detects at least one sub-step within a step, they consider it a successful detection. However, this can lead to misleading claims from vendors who boast about their 100% detection rate while ignoring the fact that they may have missed a significant number of sub-steps. On the other hand, visibility refers to the total number of sub-steps detected across all the steps. Vendors with poor visibility often focus on detection, as it is a much lower hurdle to pass, while neglecting to mention their lower levels of sub-step detection. Therefore, 100% visibility signifies a much more impressive result than 100% detection, as it indicates that all sub-steps were detected.
Another crucial aspect of the evaluation is analytic coverage. This measures an endpoint protection platform’s ability to detect sub-steps without delays or configuration changes. The quality of each detection also plays a significant role in providing useful context to security analysts when investigating an alert and distinguishing real threats from false positives. MITRE categorizes detections as techniques, tactics, general, telemetry, or none, depending on the information they provide about the action performed and the intent of the activity.
To comprehensively evaluate a vendor’s performance in the MITRE ATT&CK Evaluation, it is important to consider two additional factors: configuration changes and delayed detections. After the scenarios are tested, MITRE allows vendors to make changes to their systems and retest the entire attack sequence. This allows vendors to rectify any errors made during the initial testing. However, it is essential to know whether a vendor’s results occurred with or without configuration changes to gain a fair assessment of their performance. Similarly, delayed detections occur when the endpoint agent cannot detect a threat in real time and requires additional data to confirm the presence of malicious activity. Real-time alerts are considered more critical in real-world scenarios.
In conclusion, the MITRE ATT&CK Enterprise Evaluation is a comprehensive assessment of vendors’ ability to detect and respond to simulated attacks. By understanding the evaluation categories, how they are measured, and the factors to consider when evaluating vendor performance, readers can discern the truth behind the vendors’ claims and make informed decisions. Transparency and accuracy are vital when interpreting the results of the evaluation, allowing organizations to choose the most effective solutions for their cybersecurity needs.