What You Need to Know About the 2022 MITRE Engenuity ATT&CK Evaluations

EDR technology has grown to become the most important cybersecurity tool an organization employs, especially with the pandemic-related increase in remote work. Increasingly, this type of protection is focusing on attacker behavior rather than content filtering (anti-virus). Malware can easily be changed to avoid detection, but the tactics, techniques and procedures (TTPs) of attackers rarely change.

What Are MITRE Engenuity ATT&CK Evaluations?

MITRE ATT&CK evaluations are designed to test the efficacy of these evolving protection products, showcasing their ability in real world attack scenarios. The results provide valuable information because they are not tainted by number of customers, price or any other considerations beyond the product’s ability to identity, detect and prevent threats.

MITRE Engenuity does not declare an official “winner” in these evaluations, nor do they assign overall scores, rankings or ratings to vendors and their cybersecurity technology. The results are simply presented based on four categories of visibility and detection, offering a transparent assessment of all the detections a given security solution has produced for different stages of a specific adversary’s attacks.

The 2022 MITRE evaluations differ somewhat from the 2021 methodology, which focused on threat groups such as Carbanak and FIN7 who leverage common malware. In 2022, the tests  shifted to explore how threat groups such as Wizard Spider and Sandworm exploit ransomware—which is now the number one threat for any organization of any size.

Wizard Spider is a financially motivated criminal group that has been conducting ransomware campaigns since at least August 2018. The Sandworm Team is a destructive Russian threat group that has been attributed to Russian GRU Unit 74455 by the U.S. Department of Justice.

Why are these attacks groups notable? Wizard Spider was allegedly the originator for TrickBot and is known for utilizing Ryuk ransomware, while Sandworm is best known for NotPetya, which ripped through the internet in 2017 and took down shipping giant Maersk.

Trends Analyzed

Reviewing the MITRE Engenuity ATT&CK Evaluations shows some interesting data. Year over year, there was a 4% increase in visibility as an average for the entire pool of tested products. This shows an industry-wide commitment to MITRE and highlights its increasing importance. The pool of tested products also continues to increase, now reaching 30 products in 2022. As a comparison, VirusTotal shows 73 different endpoint vendors in total.

These 2022 evaluations show some big names in the bottom half of the list. Many of these products have been embedded in large enterprises for many years. Another interesting piece of data shows EDR providers who chose not to participate in the optional Linux environment often finished poorly compared to peers.

In security, best-of-breed products often win out over time when compared to products owned by IT conglomerates. Accordingly, Fortinet and Palo Alto Networks performed well in the 2022 testing for detection or visibility.

Overall, many of the products that rely on signatures or content analysis struggled with prevention rates in 2022. The tests were not only about the execution of malware, but also utilized techniques that try to live off the land or use existing tooling within the operating system. Once ingress tool transfer was missed, this led to compromises.

All of the new participants to MITRE testing finished near the bottom. This may be a result of being poorly aligned to the MITRE ATT&CK framework, which is considered the ruler by which we measure attacker behavior. Several of the product vendors did not test against protections and thus received scores of zero.

Notable Companies

SentinelOne

SentinelOne had two reconfigurations around network activity but was able to score highest in the area of prevention. The company has now been at the top of these charts for three years running.

Cybereason

Cybereason achieved a 100% detection rate in the MITRE evaluation. Like SentinelOne, this company has been increasingly successful in recent years, though they lack the name recognition of a company like Crowdstrike.

Palo Alto Networks

Palo Alto continues to impress, but their Participant Configuration was not submitted at the time of this writing. It is likely their testing included their firewall product as in years past, but also may have included some of the newer cloud-based technology.

Crowdstrike

Often considered the market leader by name alone, Crowdstrike has improved in this evaluation year over year. While there were several delays in detection noted this year (also seen in 2021), the company was able to reduce the amount of configuration changes throughout the test. These reductions should increase comfort that their out-of-the-box configuration will be successful for most consumers.

Microsoft

While Microsoft continues to improve its security products, this year’s evaluation did include several products beyond endpoint solutions (Defender for Endpoint, Defender for Identity, Defender for Cloud Apps and Defender for Office365). Microsoft should have a leg up with the telemetry they see globally, but they clearly rely on the aggregate of tooling. This could impact the test scenario since not some customers who use Defender for Endpoint may not be running the remaining tools in the ecosystem.

Trellix

The merger of McAfee and FireEye into Trellix could lead to an interesting future, but first prevention must improve. Overall, both companies finished middle of the pack when combining detection and prevention. McAfee did perform very well in detection, with their toolkit moving from middle of the pack in 2021 to top 5 in 2022. The test included not only the Endpoint Protection (ENS), Endpoint Detection and Response (EDR), but also the Web Gateway and SIEM solution.
 

Key Takeaways

The MITRE Engenuity ATT&CK Evaluation results are meant to be a helpful resource, providing significant value and transparency to the marketplace—ultimately, it is a measuring tool continues to force vendors to innovate and improve year over year.

Overall, the average participant in this evaluation continues to get better, which gives hope that the industry as a whole is moving in a positive direction. Also, many of the pure play or best-of-breed tools continue to win out over the platform or conglomerate options. Meanwhile, new vendors to the space have a lot of catching up to do to the leaders.

Finally, these evaluations continue to highlight a huge disparity between detection and prevention. This provide ample evidence of the value in active monitoring 24x7x365. None of these tools are perfect, so working with a partner like Inversion6 Technologies, which has invested building a Secure Operations Center (SOC) with an experienced team of security analysts and incident responders, is a crucial step to a successful cybersecurity program.