Today, MITRE Engenuity published their latest round of ATT&CK Evaluation results. The ATT&CK Evaluations team assessed the capability of 29 different endpoint security products to detect a range of common cyber-criminal tactics and techniques, as reflected in the real-world Carbanak and FIN7 campaigns. The evaluation is a non-trivial undertaking; it covers more than 20 different steps in the attack lifecycle, which include a total of more than 170 sub steps. The MITRE Engenuity ATT&CK results provide important insight into the inner workings of each security product beyond these test campaigns. We believe the ability to detect the techniques and tactics used for the evaluation can be reasonably inferred to apply to other campaigns as well, including campaigns that don’t exist yet.
In 2020, MITRE introduced a new Protection evaluation which tested a subset of the products to determine their ability to block key techniques and tactics rather than just identifying and logging them. The ability to detect malicious activity is important, but blocking is often preferred, given the sophistication of today’s cyber threats and recognition that 100% prevention over an extended period of time is unsustainable.
You might ask, “Isn’t it always better to block malicious activity?” But that’s not always the case. Blocking, especially behavior-based blocking is not always better because many behaviors that are malicious in the context of a cyber-campaign may also be common in legitimate operation. Although blocking of behavior may look good in tests like MITRE Engenuity ATT&CK Evaluations, it’s not always the answer for security products that are deployed in production environments. In some cases, even detection may result in more volume and investigative effort than value. Results of the MITRE testing of our FortiEDR product demonstrate this concept.
The Results of the Protection Evaluation
Here is one example. Test 13 replicates the actions of the FIN7 campaign, which is designed to expand access within an organization. It starts with the download of two files by PowerShell. Although FortiEDR identifies it as suspicious activity, it is still allowed to run because in certain cases like IT scripts, PowerShell is legitimately used to download files. Next, PowerShell attempts to log in to a valid account of a valid user to access an admin shared drive. So while the action is detected and recorded, it may remain inconclusive given the validity and because nothing definitively malicious has occurred. As FortiEDR continues to monitor and collect evidence, it is allowed to run. However, as soon as the downloaded file attempts to maliciously exfiltrate data via HTTPs, the blocking action takes place. The blocking takes place before the damage can occur, and afterwards and automated response works to undo the malicious activity.
FortiEDR was developed to protect data and automatically stop breaches in real time without having an effect on the system or users. Fortinet’s patented code tracing capability enables FortiEDR to analyze deep system activity and precisely identify the suspicious operations. It can block specific malicious activities such as command and control, data exfiltration, in addition to file tampering, such as ransomware encryption.
By design, this precise code tracing of system activity allows FortiEDR to be less aggressive at early stage blocking. It can continue collecting evidence and make a confident decision to block to thwart the cyber criminal. Although it is outside of the MITRE evaluation, it is worth noting that this patented code tracing technology keeps track of all the activities prior to blocking. Once malicious activity has been confirmed, FortiEDR has the ability to effectively roll back the malicious changes and return the system to a known good state without a reboot.
MITRE Engenuity: The Results of the Detection Evaluation
The detection evaluation showcased a similar design principle. This evaluation covered 177 discrete steps across 20 stages. Although we were quite pleased to have demonstrated ~70% detection of the applicable steps and /substeps, the majority of the steps that weren’t detected were by design. Aside from a series of substeps on Linux which were not in scope for our deployment, the most common reason for lack of detection was the conscious choice to date not to flag on discovery activity.
Process, system information, system network, file and directory, and other discovery is an element of sophisticated cyber attacks. However, it is an even more common activity in day-to-day business. Asset management and discovery systems, process performance monitoring programs, IT operations scripts, and other tasks routinely make use of discovery techniques. To avoid overwhelming the average FortiEDR user, we have chosen not to capture this information by design. That said, FortiEDR can be tuned to record more or less based on customer requirements.
There also is a difference in the way FortiEDR reports detection of Standard Application Layer and Standard Cryptographic protocols. Rather than describing it as a suspicious port activity, FortiEDR reports it as suspicious communication activity. Although our identification did not qualify as a detection during the evaluation, it was good to see the difference in third-party expectations.
In both cases, it’s not about the right way or the wrong way. The tests show the Fortinet way and are an excellent example of the type of information enterprise users can gain from MITRE Evaluations.
