In any industry, promotion and marketing are a vital part of the sales process, first attracting the attention of potential customers and then persuading them to opt for your product over those of your rivals. Test data has always been a key part of this, as the informed input of expert, independent testing organizations is often the clearest and most reliable guide to which products perform the best or are most suited to a given situation.
When product vendors make use of test data to promote their products, there are several opportunities to mislead or manipulate.
Firstly, vendor marketeers may try to hire a “friendly” test lab to perform a test with a specific methodology, test design or sample selection process which will guarantee a good result for the commissioning party.
In this situation, AMTSO’s Testing Protocol Standard should provide good protection. While it is completely possible for a test designed to produce a pre-determined result to be run under the AMTSO Standard and achieve full compliance, the transparency and openness that the Standard requires means any such manipulation will be clear and evident to anyone reviewing the test information.
Since other participants in the test are also likely to spot such issues, they have an opportunity to highlight them in the participant feedback the Standard demands. Some vigilance on the part of the consumer of test results is still required, but in most cases a test which complies with the Standard is more likely to be fair and unbiased, simply because the Standard removes the ability to hide manipulation behind opacity and obfuscation.
Another approach sometimes used in marketing is misdirection. An example would be submitting one product for testing by respected labs, and then using the resulting awards and accolades to promote and market a different product, which may offer very different features and levels of protection.
This may be achieved by, for example, submitting an “international” version of a product for testing, with a stronger set of default settings or components than a “local” version, then claiming any certifications earned by the international product should also apply to the weaker local one.
To avoid being misled by such approaches, readers need to pay close attention to marketing materials, and ensure that any claims made based on test data are indeed supported by that data. All respectable testing bodies should provide detailed information on exactly which products were tested, and this should align with the products the test data is being used to promote.
AMTSO holds its members to very high ethical standards, and expects all vendors, and testers, to provide the public with clear, fair, and accurate information on product quality, and to act in good faith. In cases where behavior which falls short of these standards is reported to AMTSO, we endeavor to ensure both that the instance observed is rectified speedily, and that those responsible for misleading behaviors are educated in the proper use of test data.
AMTSO considers the following actions as vendors acting in bad faith:
– Using test results obtained by one product to promote a different product, without disclosing the difference
– Manipulating or editing test data to hide failings or highlight spurious “wins”
– Drawing conclusions from tests which are not supported by the test data
AMTSO encourages anyone who notices the misuse of test data, for example in ways which appear misleading or manipulated, to report them to AMTSO by emailing [email protected] or using the form on our contact page.