It’s in everyone’s interest that antimalware tests are run on a fair and transparent basis, and this is why the industry got together 13 years ago to create a Standard for fair testing: The AMTSO Standard. It determines what a fair test should look like, including for example the requirement that antimalware vendors get insights into the test plan ahead of the test, and that the plan isn’t changed thereafter. Also, at the end of the test vendors need to be able to give feedback regarding the execution of the test. This creates transparency and opportunities for testers to understand whether they have missed anything in the test which may negatively impact the results.
The AMTSO Standard was created in collaboration between the industry’s leading testers and vendors who are members of our AMTSO community. At AMTSO, we have 55 member companies and organizations, and today we’re starting a new blog series in which we introduce our members.
Maik Morgenstern is CTO at the independent IT security institute AV-Test, a founding member of AMTSO, located in Magdeburg, Germany. Maik has been active at AMTSO since the beginning, and is currently a director on our board.
Maik, thanks for taking your time for this interview with us. Could you please explain in a few sentences what products you focus on in testing?
The focus of AV-TEST are endpoint security products like antivirus software for home users and EPP (Endpoint Protection Platform) for business users. We are covering Windows, Android, Linux and MacOS with our tests. Recently we have started to test EDR (Endpoint Detection & Response) as well. Besides endpoint products, AV-TEST is testing gateway solutions such as Secure Web Gateways or Mail Security products.
How do you assure that your tests are valid and the testing processes transparent?
Since tests can have a huge impact, we have to be very careful when designing and running a test and publishing the results. Our testing methodologies have undergone numerous discussions and updates over the years to keep up with the real world and the demands of our customers. All products are tested under the same rules and conditions on exactly the same hardware with the same test cases and so on. Everything that we do is well documented and all samples, testing documentation and log files are shared with the tested vendors so they can understand and even recreate the test result in their labs. We aim to deliver results that can be proven by data.
What challenges have you and your company faced when starting your tests?
Marketing claims of vendors make it difficult to assess which products are actually providing the functionality and quality that is required. Not all products are a good fit for all customers and not all products perform well all the time. When we started our regular testing, a lot of products were below average but customers had no way to know this without our testing. Due to our continuous testing and the pressure applied we saw an increase in the quality of products industry wide.
And what challenges do you see in using the right threat samples for a test? How do you collect samples for tests?
Nobody, neither testers, customers nor vendors, have a complete picture of the threat landscape. It is an ongoing challenge for everybody to create relevant sample sets by using current and relevant threats. In order to achieve this, we are constantly crawling the web for new threats, exchange data with third-parties and host our own honeypots to attract attackers. On top of just collecting the raw malware samples, we are also consuming threat intelligence feeds and prevalence data provided by external parties to select the most important threats for our testing.
Why did AV-Test join AMTSO and why do you think AMTSO is important for anti-malware testing?
Being one of the largest testing labs back when AMTSO was founded and today we always had an interest in good tests and invested a lot in maintaining our high quality. At the same time lots of bad tests were carried out by other parties which overall affected how testers and testing was viewed. AMTSO provided the opportunity to define basic principles that good testing should follow. It also enabled testers and vendors to discuss testing on another level than before and overall led to higher quality tests thus giving better guidance for users.
Taking a look at the next few years, how would you like to see the anti-malware testing environment evolve in the future?
I would love to see more commitment from some vendors that are currently avoiding testing. If they don’t agree with testing methodologies, they are invited to discuss those concerns in AMTSO. If they are afraid of bad results, it is about time they get their products continuously tested, as good tests will also improve the quality and performance of products thus providing better overall security for the customers.
What types of problems are keeping you and your company busy at the moment?
Threat actors and threats evolve all the time and so do security products. We are seeing new approaches to detection and protection and even new categories of products. Keeping up with this from a tester’s perspective is always a challenge. What are the threats customers are really facing? Where do they need guidance and which product tests are the most interesting for them?
This is certainly a concern for the entire industry, too. From your perspective, what is most concerning in the industry to you at the moment?
Maybe not an actual concern, but instead a wise saying that is being proven again and again: IT security will remain a cat and mouse game. While protection measures have become so good in the last years, attackers will still find a way around it and perform successful attacks. Every defender has to realize that this will remain an ongoing challenge and security is never done. Which also means that testing of security solutions will also be required all the time.
A great sentence to end on. Thanks so much, Maik, for your time for this interview!
Statements made by individuals may not reflect the views or opinions of their employers, of other AMTSO members, or of the organization as a whole.