Today we had an interview with John Hawes, the COO of AMTSO, to discuss the organization and the cybersecurity industry. John has been in the tech space for over 20 years, 10 of them with Virus Bulletin where he climbed to Chief of Operations. He has been with AMTSO since its inception in 2008. In 2015 he became the Chairman of the Board of Directors, but moved to his current position of Chief Operating Officer in 2017. In addition to AMTSO, John has been a director of Tick Tock Social Ltd. since 2017.
As AMTSO’s COO you’re very close to the organization. What do you like most about your work at AMTSO, and what’s maybe the most challenging?
I’d say the best thing about working with AMTSO is the people – there’s a fairly large group from all over the world who contribute to AMTSO in various ways, and in my role I have to keep in touch with them regularly. Many are people I’ve known for years and have hung out with in person at various conferences, meetings, and other events, but others I’ve only ever met through video calls and many more only through email. Most of our meetings have a social side where people catch up on each other’s lives before we get down to business. It’s great to have such a wide set of friends and contacts, and to see how their different situations affect their ways of viewing things.
The most challenging part over the last few years has been the inability to get together in person, having to work out how to keep things moving forward without those intense periods where everyone’s in the same room, being forced to attack an issue and come to a resolution before we break for coffee. Moving everything online has been a big learning curve but also an opportunity, and we’ve found that we get things done in a very different way – spread out over multiple smaller meetings and discussions rather than packed into the few intense days of a physical meeting or conference. Hopefully things will be back to “normal” sometime soon, but I’m sure we’ll hold on to a lot of the useful things that we’ve learned over the last couple of years, particularly our Testing Town Hall events which we’ve been running regularly for a couple of years now and seem like a settled part of the AMTSO landscape.
How has AMTSO changed since it first started? Have you noticed a cultural shift in AMTSO over the years?
AMTSO’s a pretty different organization now than when we first set it up back in 2008. Back then we were a fairly small group, all volunteers with day jobs, almost all engineers by background; we were working on a fairly limited set of problems, all related to fixing a set of known issues with the way most testing was being done at the time. Through our first five or so years we resolved a good chunk of those issues, mainly through the process of generating guidance papers, setting out what was agreed to be best practice. Just the act of agreeing on those papers turned out to be the bulk of the solution, as most of the major test labs were in the room contributing their thoughts and hearing about potential pitfalls from this very diverse expert group – I definitely learned a lot myself, and put most of it into practice in my testing work, and I’m sure other testers benefitted similarly.
Then we started work on building our Standard, which took things a step further by codifying the principles we’d spent the first few years building up and setting in place a system to track and prove that they were being followed. I think that process did lead to a change in the culture, and structure, of the organization – firstly the increase in effort going on, and outreach to other interested parties expanded our membership by a fair amount; then we realized that running the Standard and the associated compliance program was beyond the abilities of a smallish group of part-time volunteers, and we had to make AMTSO more professional. We expanded our core team beyond the Board of Directors, who did most of the active work in the early years, to include an executive team and several people like me who dedicate serious amounts of time to the organization. That led to a lot of changes in how AMTSO operates, and how much we’re able to achieve. It seems clear that these changes have significantly improved AMTSO’s value to its ever-expanding membership.
Are there any challenges you see within AMTSO and how do you intend to address them?
As mentioned above, the biggest challenge recently has been the absence of our regular in-person meetings and events. For our first decade or so we got most of our work done in semi-annual bursts of presentations and debate over a couple of days. But apart from the daytime work, there was always a huge amount going on around the fringes of the meetings, with discussions continuing during coffee breaks, over dinner and often long into the night. It was often at these informal times that the best new ideas emerged. With these events off the table for the past few years, we’ve had to find alternative ways of getting connected to members and finding out exactly what the industry needs to come together to work on. We’ve made a lot of progress on that with things like our Testing Town Halls, but getting wider engagement from our ever-expanding membership continues to present difficulties. I think we’re all very much looking forward to getting back to meeting in person with all the extra opportunities that provides.
The other big challenge, and one we’ve faced from the beginning, is getting the word out about what we’re doing. As an engineer-heavy group with very limited resources, promotion and marketing has always been a little sidelined with the organization, but as we grow and get more done, it’s become ever more important to communicate that to the world, to both those who work in or closely watch our industry and also the wider public who rely on tests to judge what to buy. In the last couple of years we’ve really stepped up our efforts in this direction, and it feels like that’s starting to bear fruit too, with much more recognition of the organization and what it stands for.
What does fair or ethical testing mean to you?
Fairness in a comparative test usually focuses on balance, avoiding bias, and making sure everyone’s treated equally on an even playing field. That can be difficult at times, and a lot of our guidance covers various things that can lead to bias and how to avoid them. The main thing though really comes down to transparency – if you’re being up-front and honest about how your test operates, it gives everyone the chance to figure out for themselves if there are any balance issues that matter to them. That’s the main focus of our Standard: opening up all the details of a test for the world to see and providing a forum for feedback, so if anyone spots a potential issue with a test then that gets pointed out. It also means that the tester learns about any problems quickly, and can address them if necessary.
What lessons from AMTSO did you bring over to your own company, Tick Tock Social Ltd. and vice versa?
Tick Tock has a number of clients and projects which are generally in some way connected to security, including being a big part of the Smashing Security podcast, and I try to keep them fairly well separated, but of course there’s some cross-pollination; a lot of things I’ve learnt from the admin side of AMTSO have proved useful elsewhere, and I’ve always found doing writing work for various news sites helps me keep on top of what’s going on in the industry, and the security world, which is useful when I’m working on AMTSO stuff, so it feels like a pretty beneficial setup for both sides.
What challenges are you faced with in your particular field?
I guess the main challenge with my current role is the diversity of skills required. There’s a lot of the behind-the-scenes stuff to do, like running our websites, managing the finances and legal things, and keeping track of all our members and their various representatives, as well as keeping in touch with potential members and other interested people. Then there are the more visible things, such as keeping our test tracking system up to date, researching ideas and finding the right people for our various project teams, drafting and editing the documents those teams produce, and of course, when the world lets us, arranging our in-person meetings, with all the venue hires, catering, and other miscellaneous jobs that come with that. There’s always a lot of very different things that need doing, and constantly having to switch between different modes of working can be challenging. Sometimes just keeping track of my task list is a pretty major job in itself.
What do have your sights set on for the future of AMTSO?
One of the main areas we’re working on expanding into is providing more targeted help and advice to the people who are the most active users of testing data – the purchasers of security solutions, mainly in business contexts. We’ve been looking into various ways of helping out CISOs, IT leaders, and other purchasers in the complex business of deciding how best to protect their users and their data, and we expect to launch some big initiatives in this direction later this year.
Alongside that, we’ll continue to focus on our existing work of promoting better testing, through our Standard and various guidance projects as well as through our promotion of what our innovative tester members are working on. There are always new techniques and methods emerging, both in malware and anti-malware, and these all need to be addressed by proper, fair and independent testing to make sure the best protections are being made available, so there will always be work to do to help move that process along. Our most recent step has been an in-depth look at the IoT world, and how testers can measure the efficacy of products aimed at protecting connected devices beyond traditional “computers” and “phones”, which resulted in our first Guidelines for Testing of IoT Security Products. Doubtless there will be many more changes like this to address, and we’ll keep working on those things as they emerge, hopefully drawing in input from the new companies that spring up in these new fields, both creating technology and testing it out.
Sounds like exciting next steps for the cybersecurity industry! Thank you John for participating in this interview with us, and all of us at AMTSO thank you for all the hard work you do!
Statements made by individuals may not reflect the views or opinions of their employers, of other AMTSO members, or of the organization as a whole.