Exploring the Results from the 2023 MITRE ATT&CK Testing Exploring

MITRE Engenuity ATT&CK® Evaluations released its latest round of independent ATT&CK Evaluations for 30 well-known enterprise cybersecurity solutions. Through the lens of the MITRE ATT&CK testing and knowledge base, this round focused on adversary behavior informed by Turla, a known Russia-based threat group. 

THE EVALUATION PROCESS 

The ATT&CK Evaluations team chose Turla based on its innovative stealth, the relevancy of its activity to various sectors and the breadth of open-source reporting on its tradecraft. The emulation represents how Turla achieves persistence with a minimal footprint through in-memory or kernel implants, evades detection by defensive tools and exfiltrates sensitive information from Linux and Windows infrastructure. This type of detection is fairly advanced and not necessarily easy to identify and stop. 

The 2023 evaluation included solutions from AhnLab, Bitdefender, BlackBerry, Broadcom, Check Point, CrowdStrike, Cybereason, Cynet Systems, Deep Instinct, Elastic, ESET, Fortinet, HarfangLab, IBM Security, Malwarebytes, Microsoft, Palo Alto Networks, Qualys, Rapid7, Secureworks, SentinelOne, SOMMA, Sophos, TEHTRIS, Trellix, Trend Micro, Uptycs, VMware, WatchGuard and WithSecure. 

MITRE Engenuity test results don't signify that you should expect 100% protection from any particular product; however, this is the best current measure of quality and accuracy for endpoint detection. There are known methods and bypasses for many of these products and some may require additional modules in order to match the tested configuration. 

While the level and quality of telemetry varied, almost all of these vendors were able to stop or prevent each test. These results should provide us with a greater sense of confidence around these tools. Many of the participants are going to analyze the data with a “100% protection coverage” mindset but they may fall short of acceptable preventive controls. 

Let’s look at the trends and data. 

TRENDS ANALYZED 

The MITRE Engenuity ATT&CK Evaluations shared some fascinating data points. Year over year, there was a 3% decrease in visibility as an average for the entire pool of tested products. The median decreased by about 5%. This is due to some new entrants dragging down the overall score. The number of products tested also continues to remain high. Once again 30 products were evaluated in 2023, showing the commitment to MITRE. VirusTotal shows 72 different endpoint vendors in total. Only Checkpoint did not have any test results, as there were noted technical incompatibilities.  

This year’s evaluations show some expected big names at the top of the list like Palo Alto Networks and CrowdStrike. We also saw Microsoft crept into the top five. 

It’s often said or seen that in cybersecurity that best-of-breed products win out over time when compared to products owned by IT conglomerates, such as Blackberry and VMware. This is interesting as Cisco no longer participated in the testing but has allegedly been in the market to acquire an EDR company. IBM, a new entrant to MITRE testing, falls into this conglomerate type of organization and the results are in line with that generalist approach. 

The tests in 2022 were not only about the execution of malware but they also utilized techniques that try to live off the land or use existing tooling within the operating system. Once ingress tool transfer was missed, this led to compromises. The 2023 tests had similar testing criteria. Turla adopts novel and sophisticated techniques to maintain operational security, including the use of a distinctive command-and-control network in concert with their repertoire of using open source and in-house tools. 

Many of the new participants in the MITRE testing finished near the bottom. This may be a result of being poorly aligned to the MITRE ATT&CK framework, which is considered the ruler for measuring attacker behavior. Several of the product vendors did not test against protections and thus received scores of zero, which affected the overall score Some of the vendors were seen with a high number of “configuration changes”, which seems contrary to the purpose and value of these tests. Changing the software and re-testing doesn’t seem to provide an honest representation of what consumers are buying. Several of the solutions are using cloud-based detection engines or signaling, which cause delays in detection and prevention.  

NOTABLE COMPANIES 

CrowdStrike 

They are the market leader with the largest brand reputation across the competitive landscape. The previous couple of years, CrowdStrike appeared to be largely focused on acquiring technology, integrating products and likely rethinking their backend after the Humio acquisition. It appears that they have executed most of the work that may have been distracting from its core competencies. No longer. CrowdStrike shined with one of the best performances in any of the MITRE testing. Their detection and quality of detection were perfect, and their prevention rating were also high. Crowdstrike does continue to have issues with detection delays, which is likely caused by some of the detection mechanisms coming from the cloud engine. 

Microsoft 

Year over year, Microsoft continues to improve their endpoint product and seemingly rebrands its name as well. This year Defender for Endpoint performed well overall, reaching its highest marks, however its quality of technique level detections was poor compared to other top tier EDR products. It should be noted that the results are not only with Defender for Endpoint, but also required Defender for Identity. Microsoft Defender suite includes Microsoft Defender for Endpoint, Microsoft Defender for Identity, Microsoft Defender for Office 365 (Email and collaboration), Microsoft Defender for Cloud Apps (Applications), Microsoft Application Governance and Microsoft 365 Data Loss Prevention. There are quite a few products to configure, maintain, administer, and monitor. 

Palo Alto Networks 

A company known for its network security has been able to make a significant splash in the endpoint space. In my experience within security, solution providers cause impact in a single area: endpoint, cloud, network, identity or application. Palo is the market leader in firewalls and has been able to consistently show very well in endpoint detection, making it a unicorn in the sense they perform well in multiple security areas. Palo was the only company to secure perfect scores across the board. 

SentinelOne 

After being on top of the results for years, SentinelOne has taken a step back. Its overall performance in prevention was still very high, it lowered in analytic detections and quality of detection. SentinelOne was busy in the pages of business magazines with acquisitions, potential acquisitions and trying to build out XDR strategy while integrating products. Their analytic score appears to be affected by “extenuating circumstances” according to MITRE. SentinelOne still received a very favorable performance in the protection tests. 

Cybereason 

This product continues to shine even through the tumultuous business environment. Once a star and potential multi-billion-dollar IPO target, Cybereason may have missed the boat on going public but continues to focus on product and mission. It has not allowed itself to get distracted with acquisitions or other things that impact their core mission. Cybereason performed nearly perfect in detection, quality of detection and prevention. 

BitDefender and Sophos 

Both of these legacy anti-virus companies have made their way back with significant year-over-year improvements. It should be noted that both organizations had a large number of configuration changes, which require a re-test. Re-test feels like an unfair advantage, because in the real world there are no “do overs”. 

KEY TAKEAWAYS 

The MITRE Engenuity ATT&CK Evaluation results are meant to be a helpful resource, providing significant value and transparency to the marketplace. Ultimately, it's a measuring tool to force vendors to innovate and improve annually.  

This testing appears to be giving the market and solution providers some incentive and it helps to move us in a positive direction. Many of the pure play or best-of-breed tools continue to win out over the platform or conglomerate options. Meanwhile, new vendors in this space have a lot of catching up to do. 

To wrap up, these evaluations highlight a huge disparity between detection and prevention, providing ample evidence of the value in 24x7x365 active monitoring. None of these tools are perfect so working with a partner like Inversion6, which has invested building a Secure Operations Center (SOC) with an experienced team of security analysts and incident responders, is a crucial step to a successful cybersecurity program.