Last week, the AMTSO membership voted overwhelmingly to fully adopt its Testing Protocol Standard, a comprehensive framework designed to promote openness and transparency in the testing process.
Over the past two years, AMTSO has been developing and trialing this first testing Standard, combining the efforts of the core team in the Standards Working Group with regular review and feedback from across the membership.
Over the last few months AMTSO has been running a public pilot of the Standard, putting several tests from major test labs through the complete process of establishing compliance with the Draft Standard, as well as gathering input from journalists and analysts outside the organization.
Here is an overview of some of the major requirements of the AMTSO Standard, and how they should help ensure tests are open, transparent and fair:
The AMTSO Standard requires that all vendors whose products are included in a test are given advance notice that a test is planned, and provided with details of how the test will be run. This ensures that vendors can review and assess test design, and provide testers with feedback on any issues which may affect specific products or vendors. As testers may not fully understand how all products operate, it often happens that a test component is designed in a way which does not properly test a particular product or type of product. Making the “Test Plan” (including detailed methodology and schedules) available in advance allows such issues to be addressed and resolved.
To support notification, AMTSO has developed a Contact List system to ensure notifications reach the right people. Anyone can join the Contact List, although most notifications will only be sent to security product vendors.
Through the Standard, we have also provided a detailed framework for designing comprehensive test plans; the Standard includes a range of details which are requirements for a compliant Test Plan, and also includes advice on other components which may be useful depending on the type of test. AMTSO provides a Test Plan template and tailored advice and guidance to ensure Test Plans are as complete, accurate and informative as possible.
The AMTSO Standard stresses the importance of treating all vendors and products being tested (“Test Subjects”, in the formal terminology of the Standard) fairly and equally throughout testing.
This particularly applies to any dispute processes a test may include. Dispute processes are where issues noted during a test, such as a product missing a malicious attack or identifying clean software as a threat, are reported to the affected vendors, who then have a chance to review them and challenge any perceived errors by the tester.
The Standard requires that any product which was not given the same access to such dispute processes is highlighted in any test reports. Being denied access to disputes where other Test Subjects did have access can have a significant impact on test results, potentially biasing outcomes in favor of those granted dispute rights. This renders any conclusions drawn from the data highly unreliable.
There are requirements for testers to disclose any other information which could potentially affect test outcomes, such as the sponsor or commissioner of a test.
Right of feedback
The AMTSO Standard provides for multiple stages of official Commentary from vendors whose products are being tested. This allows those being tested to publicly share their opinions on the design and implementation of a Test Plan. If some or many of the vendors included in a test strongly feel that it is poorly designed or carried out, readers may wish to dig more deeply into the results and available methodology details, to assess for themselves if the results offer them any reliable information.
Testers also have rights of response to Commentary submitted by the vendors they are testing, to ensure that Commentary is fair and accurate.
Fair play from vendors
Vendors taking part in tests also have duties under the AMTSO Standard. To obtain “Participant” status under the Standard, which brings with it additional rights, vendors are required to provide multiple attestations to the tester, promising to avoid any behaviour which could bias test results.
Vendors are expected to respond to both calls for commentary and tester issues in a timely manner, and must also disclose any information which could be valuable to the tester in completing their analysis of products.
Commitments from AMTSO
Alongside the development and implementation of the Standard, AMTSO has committed to providing the framework and support systems to help testers demonstrate their compliance with the Standard in an efficient and practical way. AMTSO is also hosting and maintaining the Contact List system to support proper test notification, and will offer arbitration services in the event of any disagreement between testers and the vendors of tested products.
Put all together, AMTSO strongly believes that these provisions will lead to better testing. Better testing means more useful data for readers of test reports, making us all better able to select the security products best suited to our needs. This in turn should make us all more secure.
To be sure that the test data you are relying on is fully open and transparent, check for compliance with the AMTSO Standard!