Today we interview Neil Rubenking, PCMag’s in-house security and privacy expert and a long-standing member of AMTSO’s advisory board.
Neil, what was the first antivirus product that you tested, and when?
It had to be Norton Antivirus, but I couldn’t say just when. It was back in DOS days, before Windows, anyway. I remember it named some viruses for their behavior, like the 1701 virus, which added 1701 bytes to the size of the COM files it infected. (Remember COM files?). At the time I was reviewing utilities in general, and antivirus was just another utility.
Over the past years, how has your way of testing antivirus products changed and evolved?
Well, in the earliest days all I could do was report on the product’s features, and on what it claimed to do. But PCMag is all about testing, so I gradually found things I could test. Maybe I didn’t have malware samples back then, but I could check that the product does not mislabel valid programs as dangerous (false positives). I could obtain suspected phishing URLs, verify them for myself, and test the antivirus to see if it detected them.
These days I test with real-world malware, I use malware-hosting URLs as well as phishing URLs, and I even release ransomware for testing in a closed virtual environment.
You are a legend in the space, and any provider of consumer antivirus software is eager to get into your PCMag reviews cycle and get positive ratings. Outside of threat detection capabilities, what are the specific things you look out for when evaluating anti-malware software?
As you note, threat detection is the core. Removing existing malware and preventing new infestations, that’s essential. But I also look for layered protection. Does it block access to dangerous pages? Eliminate malware during download? If it doesn’t recognize a static sample, does it detect the malware after launch by examining its behavior? I test as much as I can, and I also look to reports from the amazing independent testing labs.
Beyond that, there are a host of ancillary features that may be present as enhancements. I don’t just count features and praise high numbers, though. They have to be significant to security and privacy, and they have to work well.
Do you still have your own sets of malware samples that you test on antivirus products? How do you maintain it, or is this top secret information?
Top secret, ha! No, it’s the opposite of secret. I’ve written all about what I do to gather a new set of malware samples. It’s right here.
I don’t line up products and test them all at once the way the big labs do. I test and review products individually at the time the security companies release or update them. So, it’s important that I use the same set of samples for an apples-to-apples comparison.
On the other hand, if I kept the same samples for too long, I’d wind up testing some apps twice with the same collection, which could give them an unfair advantage. Balancing those two needs, I round up a new set of samples once a year, in early spring. I’m literally in the process of doing that right now.
What role do independent testing institutions like AV-Test, MRG Effitas, and SE Labs play for your reviews?
Their lab test reports are invaluable, in two ways. First, the fact that a product even appears in the results means that the lab team found it important enough to consider, and that the antivirus company considered this kind of testing valuable. Second, of course, a set of high scores from multiple labs is excellent evidence that the product is effective. Any time I find that my hands-on results don’t align with the lab reports, I give more weight to the labs.
You have been on AMTSO’s advisory board for many years. From your perspective, which impact does AMTSO have on antimalware testing?
I wasn’t at the AMTSO founding meeting in Barcelona, but I hear it was fraught. Significant tension between testers and vendors, and a few declarations that the organization was doomed to fail. Clearly it didn’t fail!
I was privileged to attend a planning meeting of the Board and Advisory Board in Canterbury a few years later. That meeting hashed out a lot of what AMTSO is now, though it took a while to grow into the model we dreamed of. I’ve heard rumors that the brainstorming pages from the easels in that meeting are stored in an AMTSO crypt somewhere.
One big thing that came from that meeting was a strong understanding that AMTSO needs a set of standards. Now that we’ve really codified those standards, I think the biggest impact is lowered friction between vendors and testers. Everything is clear, and there’s a process for handling disagreements. The standard and guidelines also help newcomers get into testing without having to make all the same old mistakes.
Looking at consumer safety, do you feel consumers today are more, or less secure than ten years ago?
My view is that consumer cybersecurity is in balance, always about the same. If a new type of threat emerges and affects consumers, two things happen. First, some people (not many) spring into action to better care for their security. Maybe they sign up for an identity theft recovery service, or upgrade from antivirus protection to a full-on suite. Second, the security companies work to counter this new type of threat. For both those reasons, we swing back to a balance.
While consumer security stays about the same, I think consumer privacy is more endangered than 10 years ago. Hardly a week goes by without a new data breach. And the breaches pile up—once your data is exposed, it stays exposed. There’s nothing you or I can do about a breach that happens because someone, somewhere made a mistake. All we can really do is watch for signs that someone is playing fast and loose with our data, and head off any attempts at identity theft.
What are the next big cybersecurity issues you think consumer safety companies should watch out and prepare solutions for?
I’m not a fan of predictions. I receive a lot of predictions in email, and I carefully archive them to my predictions folder, unread. I don’t imagine I can know what the next big thing in cyber danger will be. The important thing is that researchers aiming to maintain protection for consumers spot that next big thing the moment it appears, if not sooner, and get to work counteracting it. And of course, the testers must invent ways to verify that the new techniques actually work.
Statements made by individuals may not reflect the views or opinions of their employers, of other AMTSO members, or of the organization as a whole.