In continuing our interview series with different AMTSO members, we sat down with Simon Edwards of SE Labs to discuss the company and its partnership with AMTSO. Simon is one of the founding members of AMTSO and has served on the board of directors since 2016. In the past, Simon has worked as an IT journalist, a Technical Director at Dennis Technology, and in 2015 he founded his security testing company, SE Labs.
Simon, thank you for answering our burning questions today! Can you explain what products you and your team focus on in testing?
We test a wide range of security products and services that cover endpoints, network detection and protection, and cloud services such as email and web security gateways.
When testing security products, what are the particular procedures that your company goes through?
One critical part of how we test involves learning how the criminals and other attackers behave in the real world. We then repurpose their tactics, techniques, and procedures to create realistic attacks for use in testing. We don’t simulate attacks but run full attacks in the same way as the bad guys.
It’s super interesting that you don’t simulate exact attacks- can you explain more on how your full attack is different from a simulation?
Attack simulations take shortcuts. To generalise, they might generate automated network traffic that looks like a compromise has happened or is happening. Their advantage is that they allow fast testing with lots of test traffic, and without a team of expensive hackers working consistently over a long period of time.
We want to produce the highest quality testing so we take the expensive option, using our well-trained security testers to run real attacks. Simulated attacks can provide evidence that products do or don’t work. However, non-simulated attacks, such as we run, provide more convincing evidence because our attacks are much the same as those run by real attackers.
Since you strive to not to simulate your attacks when testing, are there particular problems that the SE Labs teams face when trying test products? And how do you try and resolve these issues?
The problem is that in much security testing there is an element of simulation, where the testers have to compromise realism in order to get a large amount of work done. The less realistic a test becomes, the less useful it is in assessing products. By testing like hackers we avoid that issue, although it means we put quality over quantity. In our tests you won’t see millions of attacks run against a product, but you will see the results of our red team testers genuinely hacking through various security products in the same way that real attackers work.
In your testing, what challenges do you see in using the correct threat samples? How do you collect samples for tests?
We don’t collect malware samples and scan them. Because we run realistic attacks, our main challenge – when it comes to collecting ‘test samples’ – is to create attack scenarios that are relevant. To achieve that we need to keep a close eye on the very latest threat intelligence. That’s why we say we perform ‘intelligence-led testing’. While it might be exciting to learn about the latest developments in hacking, in the real world hackers use tried and trusted techniques when they can get away with it. So it can be tempting to load up a test with really cutting-edge techniques, but we need to also test using more well-known methods because there’s plenty of that bad behavior occurring on the internet right now.
Why did you decide to have SE Labs join AMTSO and why do you think AMTSO is important for anti-malware testing?
AMTSO’s primary goal is to improve security testing. By being part of the conversation we can bring our own experience of advanced red team testing into the mix. We’ve always been a big advocate of transparency and that is another major function of AMTSO – to recognize openness in testing. That’s why every one of our endpoint protection tests has complied with AMTSO’s testing standard since its introduction at the beginning of 2018.
Transparency is indeed a hugely important aspect in the cyber world! Thank you for taking the time to do this interview and for sharing your insights with us today, Simon.
Statements made by individuals may not reflect the views or opinions of their employers, of other AMTSO members, or of the organization as a whole.