Latest Press Release
AMTSO Membership Approves Major Step Forward in Testing Standards
San Francisco, California – May 22, 2017 – The Anti-Malware Testing Standards Organization (AMTSO) moves the first ever set of official Standards for anti-malware testing forward to the next level – the development of a full accreditation and endorsement program.
Several recent controversies in published tests have highlighted the need for objective standards to measure the quality and fairness of tests conducted on security products. AMTSO’s Standards project is designed to provide a framework to enable unbiased evaluation of testing approaches and methods.
“The Draft Standards aim to provide guidance to testers on how best to ensure that the testing of anti-malware solutions is open, transparent, and fair,” said Dennis Batchelder, President of AMTSO and CEO of AppEsteem. “Tests should provide useful and actionable intelligence for everyone who relies on test data, from analysts and media to purchasers of security products, as well as vendors themselves trying to improve their products. These Draft Standards are designed to encourage and facilitate that clarity and fairness.”
“The AMTSO Standards will give testers protection against accusations of unfair practices and poor testing,” said Simon Edwards, Chairman of the Board at AMTSO and Director at SE Labs. “By ensuring that all stages of the testing process are clearly described, well documented and open to comment and feedback, testers are given a strong position to back up the claims made in their test reports, and AMTSO is committed to supporting and defending high-quality tests that follow the Draft Standards”.
Since its formation in 2008, AMTSO has leveraged the combined expertise of specialist testers, developers of security products and academics working in the security field to develop a set of guidelines and resources to promote better testing.
In 2016, the organization formed a Standards Working Group (SWG) to drive the development of a definitive set of industry standards. The SWG regularly sought comment from AMTSO’s membership and the wider security community. During the latest round of feedback in the last few months, over 200 comments and suggestions were collected and absorbed into the final Draft document.
“The Standards Working Group has put in a huge amount of work building out these Standards,” said John Hawes, COO of AMTSO. “We’re very proud of how far we’ve got in little over a year, and we are looking forward to getting the structures and processes in place to ensure the Standards are implemented and adopted across the industry.”
At a recent AMTSO meeting in Kraków, Poland, AMTSO members adopted the Draft Standards and authorized the Standards Working Group to proceed with developing the procedures and operations to implement the Draft Standards. Over 60 delegates representing 35 member organizations were present for the discussion and voting.
“The WannaCry outbreak had every security vendor working extra hours to demonstrate their product stopped it, which shows the importance of independent testing to verify vendors’ often bold claims,” said Martijn Grooten, Editor of Virus Bulletin, an AMTSO member organization since 2008. “I was pleased to see the AMTSO Testing Standards officially receive Draft status, helping testers adopt clear and transparent procedures.”
“In an industry as fast moving and increasingly technology diverse as endpoint security, there is a strong need for standard test methodologies to help consumers and businesses determine what products are best for them,” said Norm Laudermilch, CISO at Sophos. “AMTSO’s vote to approve Draft Standards is a monumental move toward producing fair, consistent, and quantitative metrics to drive those decisions.”
AMTSO comments on recent testing issues
San Francisco, California – February 18, 2017 – Testing products in a fair and balanced way is very difficult. Product developers routinely make bold claims about the capabilities of their products. AMTSO supports the right of testers to put these claims to the test, to provide independent validation of their accuracy (or otherwise).
Since our foundation, AMTSO has strongly advocated realistic testing methodologies, exercising all features and layers of all participating products and giving genuine insight into how well they perform at stopping the types of attacks affecting real users in the real world.
We have been asked to explain AMTSO’s opinion on recent privately-commissioned anti-malware tests. We believe that we will be in a much better position to provide useful feedback on specific tests after adoption of the AMTSO standards, but we do want to respond to these inquiries by providing some general statements about problems we’ve seen in recent tests.
- We reject turning off product capabilities while comparing the capabilities of products in real-world use, as we believe that this introduces bias in the results.
- We believe that any claims about what the results of tests show must be valid and accurate, and they must provide both data and evidence that the scenarios tested and the methodologies used do in fact match the resulting claims. In our opinion, test reports without this data and evidence should be rejected.
- We believe that tests that don’t give the tested product vendors an opportunity to engage and to comment on the approach or to validate their configuration are unfair.
- We believe that all comparative tests should follow our draft standards.
- We support the rights of a tester to run any test it wants to, and to test any available product without limitation, consistent with the AMTSO draft standards.
AMTSO is currently running an open pilot to trial the draft standards, and welcomes participation from both member and non-member testers. For more information contact email@example.com.
The Anti-Malware Testing Standards Organization (AMTSO) calls for public participation in creating standards for certification and compliance
San Francisco, California – September 6, 2016 – The Anti-Malware Testing Standards Organization (AMTSO) is announcing a call for public participation in developing standards for testing anti-malware products and solutions. These standards will be the basis for AMTSO’s certification program, which is anticipated to launch in the second calendar quarter of 2017.
Anti-malware testing is the critical link between the vendor and end user. If done well, testing can independently verify whether anti-malware solutions work as vendors claim by showing which solutions are most effective at protection. However, improper testing can create misleading results, leading customers to adopt inadequate protection that risks their privacy and security. Dennis Batchelder, General Manager of AMTSO, said, “This is a great opportunity for people outside of AMTSO and even outside of our industry to get involved and help us get our testing standards right. We’d love to see participation from interested individuals, consumer advocates, academics, and publications.”
The Standards Working Group (SWG) will work in cooperation with AMTSO’s membership and other industry participants, and will meet weekly starting September 14, 2016. The SWG will hold an in-person meeting on October 24, 2016, during the AMTSO Member Meeting in Málaga, Spain. Participants may also attend this meeting by conference call. For information on this meeting, please send an email to firstname.lastname@example.org.
The first draft of AMTSO’s testing standards will be available for public review and comment on AMTSO’s website on September 12, 2016. This draft will be updated regularly and comments are invited from all interested parties. AMTSO members will vote on initiating a pilot program to test the standards at the October member meeting, and will consider final approval of the standards and certification program in the second calendar quarter of 2017.
Those interested in joining the SWG or who would like more information, please send an email. Participation is open to both AMTSO members and the general public.
About the Anti-Malware Testing Standards Organization (AMTSO)
AMTSO is a California-based non-profit mutual benefit corporation founded in 2008. AMTSO’s mission is to improve business conditions related to the development, use, testing and rating of anti-malware solutions. AMTSO membership is open to industry-wide academics, reviewers, testers and vendors. Additional information regarding the organization, including charter documents, membership and educational materials are available on the AMTSO website.
M: +1 650 867 2975