The Anti-Malware Testing Standards Organization (AMTSO) was formed in 2008 as an industry organization dedicated to improving the reliability and transparency of anti-malware testing. Since that time, we have leveraged the expertise and resources of our industry-wide membership to provide advice, guidance, and assistance in the design, implementation, and communication of testing practices.
Over the past two years, we have focused on the development of the industry’s first anti-malware product testing protocol Standard. The final Standard, which was adopted by our membership in May 2018, aims to create a framework to ensure that anti-malware product tests are run fairly and transparently, and that the public is provided with the information needed to properly assess the quality and relevance of a test.
We believe that it is valuable to the industry and consistent with our mission to review testing methodologies that do not aim to comply with the Standard. This insight can help demonstrate the usefulness of the Standard. However, the AMTSO Testing Protocol Standard is voluntary, there is no requirement that any AMTSO member, or any other party, follow it.
In Spring 2018, we evaluated an Endpoint Protection Test against the then draft AMTSO Standard to determine whether application of the Standard would have improved the accuracy and reliability of the test. We directed an independent team to complete a thorough comparison of the test and report the results back to the Standards Working Group, which was finalizing the draft Standard.
After a thorough analysis, our review team came to the opinion that if the testing lab had followed the draft Standard, the test report would have been released with higher accuracy, and consumers of the test report would have had a better indication of the effectiveness of the various tested solutions and more transparency into how the test was conducted. The review team made three primary determinations in developing this opinion:
- The review team found that the tester failed to properly notify all tested vendors in advance of the test. The Standard states that notification must include a reference to the test plan, and all tested vendors must be notified of the upcoming test. Based on the finding that the tester did not properly notify all tested vendors, some solutions were tested at a significant disadvantage, which increased the risk of the tester presenting inaccurate results to the consumers of the test report.
- The review team found that the tester failed to make it clear in the test report and summary graphs which vendors had the opportunity to dispute the test results and which ones did not. The Standard states that any material statistical differences must be called out. Based on the finding that the tester did not call out these differences, certain solutions had significant advantages over the other, which ran the risk of presenting inaccurate results to the consumers of the test report.
- The review team believed that the information provided by the tester on the report product ranking grid could be misleading to the reader, as it positioned an untested vendor with a text explanation, but with an indicative red arrow. This factor would increase the risk of consumers misunderstanding that the vendor was in fact untested.
The review team conducted its analysis with participation from the testing lab, and several vendors. During this process, the testing lab provided valuable feedback on the Standard, and we will continue to work to improve its clarity and usefulness.
We expect to see the first tests reaching full compliance with the AMTSO Standard in the next few weeks, and will continue to strongly encourage all testing organizations, and other parties engaging in anti-malware testing, to follow the AMTSO Standard wherever possible. After significant effort from our members and participants, we believe that the Standard, and the related openness and transparency we advocate, should lead to better testing, better products, and a more secure world.
– Dennis Batchelder, AMTSO President