Luis Corrons is our co-chairman of the board, and today he is our guest in our member interview series. Luis has been with AMTSO since the very beginning. In fact, the very first meeting of our organization was held in Bilbao, Spain, Luis’ hometown. Only a few months later, the second meeting took place in the Netherlands, where the organization was finally called AMTSO.
Luis, you joined AMTSO first during your work at Panda, and stayed with us when joining Avast. How, from your perspective, has AMTSO evolved throughout the years?
AMTSO started because there was a serious problem with antimalware testing. There were a few labs performing serious tests, and at the same time there were a number of people showing up that didn’t really know anything about malware and security. On top of that, security technology was evolving rapidly (behaviour detection, cloud, etc.) and tests were ignoring this, just running static scans of dormant malware.
The misinformation was disastrous for everyone: users, testers and vendors. We were aware that anyone could do what they want, we live in a free world and there is no way to stop anyone from doing a crappy test. But what we could do is at least provide information and tools to help people interested in the topic to improve their job. Getting together as AMTSO also ended up helping both testers and vendors to understand each other, as we had to work together, hand in hand, to go through this process.
Once that big step was achieved, we had to figure out what we should do next. The IT security industry evolves really fast, and after trying different approaches we decided that a Standard to test antimalware solutions was needed. Most of the issues involving testing were caused by a lack of information, and with a standard setting things straight we could address the main problem. Building this standard was going to be a titanic effort.
Since the beginning, we were just an organization formed by volunteers, and if we wanted to do an outstanding job we needed to professionalize it, hiring staff that could run the day-to-day operations. That’s why we hired our COO John Hawes and our Standards PM Scott Jeffreys. After a lot of work and effort, the AMTSO Testing Protocol Standard was born.
How does this Standard help users at all? It is not mandatory for anyone, there are testers who are AMTSO members that are not even using it.
Transparency. That’s the key word. A user that consumes this kind of test will look at the results, of course, and he will also have the possibility to check how the test was actually performed. Vendors participating can leave comments, it’s all there in its “nudity”, that’s the beauty of it.
And it is and will always be voluntary. It is true that at some point users may start demanding that antimalware tests are run under the AMTSO Standard, same as you can see how some analyst firms are already recommending it. They really want to know all the information about the test so they can form an informed decision. Some testers have told me they have increased their business thanks to following the AMTSO Standard, as they’ve got new customers looking for it.
It helps testers, even if they are unknown, as they can show how good they are at testing. It helps vendors too, as long as they are willing to play with all cards on the table, otherwise they’d rather not participate.
If it is all that good, why aren’t all testers using it?
There are tests where the Standard doesn’t make sense. For example a private test, commissioned by a vendor, that sets specific rules, like not informing the other participants, not allowing other vendors to dispute misses, etc.
There are testers that already have a strong reputation and feel they do not need to follow the Standard, as they are already following it -albeit not officially- as in general they cover most of what is required by it.
And of course, transparency is a double edged sword. Imagine you are running a test and it turns out that half of the malware samples used in the test were not even malicious. Even if the testing is flawless, the image of a tester not being able to select malware samples, or as some vendors have told me in the past “we are the QA of this tester’s malware testbed” would be negative. Take into account that if there was no dispute process, vendor “A” could have a 80% malware detection result, and after the dispute process this turns into a 99,99%… no matter how good the test methodology is, the testing company image could be seriously damaged. They’d never go for a Standard-based test unless they fix their issues.
You as a chairman certainly have a high level perspective on AMTSO’s work, while also being involved hands-on in meetings and working groups. How do you see the group dynamics between the different vendors and testers? At AMTSO, competitors get together to discuss topics that are in everybody’s interest – but different vendors have different opinions, how challenging is this from your point of view?
While I am not the biggest fan of the word “consensus”, it is something we try to reach by working hard with each other. Every task, working group etc. that we set up is made to solve a set of issues, we may have different views on how they can be addressed, but we agree on the problems, which is always helpful. Most representatives have a heavy technical background, that really favours reaching solutions based on logical approaches.
It is true that in some industries what happens here couldn’t even be imagined. But we feel we are on the same side, maybe that’s what makes our situation different. We are all fighting cybercrime and protecting users, one way or the other, being a vendor or a tester. We all work towards improving protection. Of course vendor “A” wants to be better than vendor “B” and tester “A” wants to be better than tester “B”, that’s life, but here at AMTSO we are mainly technical people fighting the same bad guys, and collaboration has been going on for years, way before AMTSO was created.
The first version of the Standard was established in 2018, 10 years after the organization was created. What was AMTSO working on before deciding to go for the standard?
The first work, really important, was creating documentation; from “The Fundamental Principles of Testing” to different guidelines and best practices. This is information needed for anyone that wants to be involved in anti malware testing, covering different topics going from dynamic testing, sample selection, false positives, performance testing, etc.
Afterwards, we created the AMTSO Security Features Check Tools, free for every user or organization. It allows them to check if mobile or desktop security software is working properly, being able to detect malware, connect to the cloud, block phishing, etc.
In which way is AMTSO important for Avast, why is Avast a member?
To give you an idea how important AMTSO is for Avast, I remember Pavel Baudis, co-founder of Avast, attending the different AMTSO meetings when it was created. When I joined Avast, I was a member of the board at AMTSO and I asked them if they wanted me to be there after I joined the company, and the answer was a resounding yes.
Avast is involved in several working groups and it belongs to the group of most active members. Tests of our products are essential for us to have an external benchmark which helps us to continuously improve our products, and show our strong results to our users. Of course, tests only are helpful if they are profound and fair, which is why AMTSO has been very valuable for us.
Do you see the need for AMTSO to move into fields beyond AV testing, and which fields would be relevant?
Well, antivirus or antimalware testing were the easy words so people could recognize what we were working on, but our vision goes beyond that, we really are focused on IT security testing, that includes many more platforms and ecosystems than just “antivirus”. I have explained what drove the creation of AMTSO, as security technologies and threats are continuously evolving, there are issues that have to be addressed, and we try to foresee what issues we might encounter in the future and what we could do to address them. For example we already have a group working on security testing around IoT for quite a while.
What might be the challenges for AMTSO in moving into new fields?
The main challenge might be to attract talent from different leading companies in the different fields. Most of the vendor members in AMTSO are what we could call “traditional” vendors, well known companies that have been working in security for decades. But there are fields with new startups, or fields that are covered by different types of vendors or even testers that we need to attract.
Thanks so much for these insights, Luis. Is there anything else you’d like to add?
As I have been involved in AMTSO since the beginning I’d like to take this opportunity to express my gratitude to a few individuals that are no longer participating in AMTSO but without whom we couldn’t be where we are today:
Jaimee King, without her, we would have never been able to form the organization. She was the one that helped a bunch of nerds to create AMTSO from scratch, and for many years she offered invaluable guidance and advice.
Matt Williamson, whose passion was a driver for us all, and although he was with us for a brief period of time, he really made a difference and won’t be forgotten.
Karel Obluk, former CTO at AVG was also heavily involved in AMTSO and helped us grow and mature.
Righard Zwienenberg, first with Norman, later with Eset, was at the heart of the organization from the beginning, too. Impossible to remember the number of meetings that have been a success thanks to him, not to mention AMTSO itself through the different roles he has had in it such as CTO, President and member of the board of directors.
Mark Kennedy, with us since the beginning and for many years in the board of directors, he always defended his ideas with passion and logic, bringing his fabulous experience to the table.
Stuart Taylor, without whose guidance during the first years as chairman of the board we couldn’t have accomplished our goals.
What a nice way to end on, showing the great team effort behind AMTSO. Thank you very much for your time, Luis!
Statements made by individuals may not reflect the views or opinions of their employers, of other AMTSO members, or of the organization as a whole.