On November 10th 2021, AMTSO held its most recent Testing Town Hall where speakers of SE Labs, MRG Effitas, and Secure IQ Lab presented and discussed topics focused on the anti-malware and cybersecurity world. It was an opportunity for member and non-member vendors and testers to engage with the cybersecurity community and gain inside knowledge.
NDR Testing
Simon Edwards, founder and CEO of SE Labs, spoke about how his company performs NDR tests on network security appliances and how the company goes through network detection and response. Edwards explained that SE Labs tests because they want to help improve vendors’ products, help security customers be able to choose the correct product for them without bias, and help with cyber insurance risk analyses. Even though SE Labs has been working on NDR tests within the network space for a long time, Edwards pointed out that most tests haven’t been published because many products still have much improvement to make.
The testing environment for NDR testing consists of products that are designed to sit in large networks, like a data center, which is why these types of products are usually bought by banks and ISPs. The first stage of this type of attack chain starts with either a phishing email or malicious PDF that contains an exploit in it. Next the attacker will start to sniff around in the system to try and escalate their privilege to record keystrokes or capture screenshots. It’s important to note that network detection response product won’t necessarily detect the same things that you would in an endpoint product. For example, escalating privileges on a Windows virtual machine would not generate any interesting network traffic to be detected. Edwards stated that when SE Labs goes to test a product, they look at what known malicious attackers have done in recent months and years and they generate variations of attacks based on that.
When testing a detection product, SE Labs has multiple levels that the product needs to pass. Edwards showed an example of a test case instance and explained each point of what they look out for. Did the product detect the attack at all? Did the product detect the attack as it was first delivered? Did the product detect the execution of the attack? (To get a good score from SE Labs, the product technically only needs to pass either detection of delivery or execution, but detecting both is good too.) The detection of the attackers listing files on the PC and the detection of privilege escalation are listed as N/A because they would not expect a network detection product to be able to see those things anyway. However, a network detection product should still be able to see the behavior that is exhibited after a privilege escalation, lateral movement, and lateral action.
After going through how SE Labs runs their tests, Edwards gave his opinion on testing in general. He believes that bit-for-bit similarity and “100% apples to apples testing” is impossible with any kind of product. This means that it’s entirely impossible to create a test that would be 100% the same as its real-world attack since each time an attack occurs, even if it is the same type, hashes of files and other details won’t match between each attack (which is perfectly normal). He comments that even attackers do not attack 100% in the same way every time – there are always variations. However, he believes that “close parity” is possible and that it’s all that can be done in the world of testing. The important thing for testers and vendors isn’t to replicate their products and tests 1:1 to handle and be exactly the same as an attack, but to be the most useful product/test to the end user.
Finally, Edwards announced that SE Labs would be releasing their Annual Report for 2021 newsletter for security vendors in the coming weeks. To sign up for their newsletter, visit https://news.selabs.uk/vendor
Smartphone privacy
Zsombor Kovacs, Research Director at MRG Effitas, presented about the intersection of privacy, smartphones, advertising, and political elections. Kovacs explained that Cambridge Analytica, a now defunct data broker company, was able to influence not only the Brexit and United State 2016 presidential elections, but other international elections and referendums, would lobby for oil companies, and would help spread the belief of climate change hoax. They would purchase and use massive amounts of data from Facebook to showcase convincing advertising to the average user. Kovacs said that the technology that was previously available only to politicians or big companies, through brokers like Cambridge Analytica, is now readily available to almost anyone. Facebook now offers targeted marketing strategies to any performance marketers- instead of relying on search interests based on cookies, Facebook can aid you in targeting real individuals who might be interested in your products. Kovacs draws the conclusion that aggregated data is power, not just in marketing and in selling things to people, but in virtually everyone’s everyday life.
The presentation was then redirected towards Apple and Google, and how they are marketing themselves as a defender of privacy. Both companies recognized that user privacy was a lucrative selling point for their phones. However, the fallacy in this is that many apps in both in the Apple and Google Play stores contain an app analytics framework provided by Facebook. Because of this, Facebook can still track you, regardless if you have a Facebook or Instagram account or not and regardless if you have opted out of tracking in your phone’s settings. Kovacs also showcased articles that discovered that contrary to their advertising, opting out of app tracking on iPhones was a moot point: “Transparency made no difference in the total number of active third-party trackers, and had a minimal impact on the total number of third-party tracking connection attempts.” Google’s stance in similar to Apple’s, but more analysis into Android’s privacy on devices has yet to be released.
Kovacs concludes that even with these tech giant’s stances, the problem of selling user data still persists. He found that the general public still demands privacy and wants to deny tracking on their devices. He suggests that the AV community could fill this gap and create a potential “privacy guard” that protects users from app tracking and it would be a way for AV companies to get into Apple’s App Store since antivirus software apps are not allowed.
Cloud WAF Testing
David Ellis, VP of Sales and Corporate Relations of Secure IQ Lab, centered his presentation around some of the results from Secure IQ Lab’s testing of Cloud Web Application Firewall technologies. The test included products from AWS, Barracuda, Cloudflare, F5, Fortinet, Imperva, Prophaze, SiteLock, and StackPath. Ellis stated that this iteration of his company’s testing was mainly to create a baseline for WAF technologies going forward, and that cloud WAFs have evolved so much that most vendors tested do not require tuning. He also pointed out that cloud WAF providers are starting to enter into the security space, and WAF technology still has opportunity for improvement.
When testing WAF technologies you have to consider the product’s work flow, uses cases, and what’s important to enterprises and the businesses that use the product. Ellis says that the reason why WAFs are so important is that they are one of the most common breach vectors. Since businesses are trying to constantly update and create new applications and code, security is not at the forefront of how they get to market. Secure IQ Lab’s test report included 9 successful vendor tests, ran multiple test cases, and ran over 22,000 attacks. There were two score results- Security Efficacy Tested and Operational Efficiency Validated. The average Complete Security Efficacy score was 62% and the average for Operational Efficiency was 81%.
The good news out of this test was that even though Ellis’ team assumed that there would be a lot of false positives and issues with layer 7 DoS attacks, the vendors’ products as a whole were good at stopping these kinds of concerns. The areas for improvement that Ellis’ team recognized were products’ handling of HTML injection attacks, server-side template injection attacks, path traversal attacks, and custom web shell attacks. Ellis hopes that this will be the first of many tests that will help advance the security of WAF technologies.
Next event
Slide decks and call recordings for this event are available to AMTSO members only. To find out more about becoming an AMTSO member, see our joining page. The next Testing Town Hall will be on January 12th, 2022, and is open to all AMTSO members and the public. To register your interest in attending our next Testing Town Hall event, please contact us at [email protected] or complete the event registration form.