Testing Town Hall January 2021

On January 13th 2021, AMTSO held its most recent Testing Town Hall where speakers from Cybereason, AV-Test, and NetSecOPEN presented and discussed topics focused on the anti-malware and cybersecurity world. It was an opportunity for member and non-member vendors and testers to engage with the cybersecurity community and gain inside knowledge.

Insights into the MITRE ATT&CK framework

Sam Curry, CSO at Cybereason, presented how vendors view the MITRE ATT&CK framework and evaluations. MITRE ATT&CK is “a globally-accessible knowledge base of adversary tactics and techniques based on real world observations.” Curry explained that to leverage MITRE ATT&CK correctly you must understand your current security state, plan SOC improvements, and define your desired state. MITRE ATT&CK is typically used for adversary emulation, SOC maturity assessments, cyber threat intelligence enrichment, red (and purple) teaming, behavioral analytics development, and defensive gap assessments. This can be implemented in five steps: Establish Inputs, Create an Adversary Emulation Plan, Run an Attack Simulation, Alert Hunt and Report, and Process and Technology Improvement. MITRE ATT&CK Evaluations entail three separate evaluations, where each simulates a real world scenario. The evaluations are focused on detections, cover a subset of the full ATT&CK framework, do not contain bias or financial factors, and there are no scores, rankings, or qualifiers – only detection types are awarded. Round 1 evaluations simulate APT3, Round 2 simulates APT29, and Round 3 simulates Carbanak/FIN7. This type of testing is useful for unbiased, technical insight into product capabilities, for POCs, and for self-assessments.

AV-Test’s approach to APT Testing

Maik Morgenstern, CTO of the AV-Test Institute, presented about how AV-Test approaches APT Testing. While AV-Test also tests XDR, MDR, and MSSP products, this presentation focused on APT Testing with EPP and EDR. Morgenstern reported that even the worst products tested by AV-Test still delivered good enough protection against malware, with most anti-malware software delivering good to very good protection. Among 334 0-day malware attacks tested on EPP products between September and October 2020, the current industry average of protection was 97.9%. However, recently APTs have become more prevalent in attacks against governmental organizations and other companies. Morgenstern shared that AV-Test started to work on APT Testing in 2018, in 2019 they had their first internal tests, and the company’s first public tests will be available in 2021. To test against APT attacks with their EPP products, AV-Test simulates a company network with different clients and servers, they set up their own C&C infrastructure, and they perform full attack chain testing. Morgenstern also noted that they describe their attacks in MITRE ATT&CK framework terms. When evaluating EPP products, Morgenstern clarified that AV-Test focuses on protection, and asks “Which stage is the attack detected and blocked?”

For the future, AV-Test plans to run and publish APT Testing for EPP products in parallel to their monthly certification tests. Morgenstern assured that after testing, vendors will receive plenty of information, including log files, screenshots, and time for feedback. For certification, “products are expected to block a certain number of attacks and don’t over block legitimate actions.” When testing EDR products, AV-Test focuses on the detection and response, and asks, “Which steps of the attack were detected and how were they reported?” Detection and evaluation of an EDR test can be rated as one of three types: Telemetry, Alert, and Technique. Morgenstern announced that for EDR products there will be two public tests a year. To wrap up the presentation, Morgenstern spoke about some things to consider – currently there is no single fair and complete APT test out there, but there are still good tests. However, no one can answer the question “Which product will protect me from any and all APT attacks?” Other useful information that was pointed out was that EPP products might not protect you from all future attacks, but it will increase the cost for someone to attack you. For EDR products, better and broader coverage means that it will be harder for an attacker to go undetected, increasing the chances of detecting a breach early on. Interestingly, if you use a good EPP or EDR product, attackers might go for an easier target instead, or if they specifically want to attack you, it will cost them more money and will be more difficult to stay undetected.

Network security testing standards project NetSecOpen

Brian Monkman, Executive Director of NetSecOPEN, spoke about his company and what they want to achieve in the cybersecurity realm. NetSecOPEN’s mission is to develop open, standardized tests based on real-world network conditions. So far, they have completed standardized testing requirements for network firewalls and network IPS. Current membership consists of test tool vendors, test labs, product vendors, and enterprises from the US and abroad. Monkman listed NetSecOPEN’s benchmark working group’s standard contains 9 test categories: Throughput Performance with NetSecOPEN Traffic Mix, TCP/HTTP Connections per Second, HTTP Throughput, TCP/HTTP Transaction Latency, Concurrent TCP/HTTP Connection Capacity, TCP/HTTPS Connections per Second, HTTPS Throughput, HTTPS Transaction Latency, and Concurrent TCP/HTTPS Connection Capacity. All approved tests must go through a rigorous certification process and all details pertaining to each test must be made public for viewing.

The next Testing Town Hall will be on April 14, 2021, and is open to all AMTSO members and the public. To register your interest in attending our next Testing Town Hall, please fill out our event registration form. To find out more about becoming an AMTSO member, see our joining page.