Test Lab: MRG Effitas Logo
Test Title: MRG Effitas Q4 2018 360 Degree Assessment and Certification
AMTSO Test ID: AMTSO-LS1-TP005

  • Statement from Test Lab:

    “A first-of-its-kind test that covers all angles, our pioneering 360 Degree Protection Test targets the key threats faced by internet users. In each test case we employ the full spectrum of Early Life Malware. We use a Time-To-Detect metric to measure how long it takes each application to detect and neutralize missed threats.”

    Test Plan

    Participants: 7
    These Vendors chose to adopt Participant status under the AMTSO Standard, gaining certain guaranteed rights in return for attestations

    “Included” Vendors: 2
    These Vendors did not chose to adopt Participant status under the AMTSO Standard, but may have engaged with the test lab in other ways

    Compliance Status: Confirmed compliant with AMTSO Standard v1.1

    Published test Report

    Publication Date: 2019-02-10

    Participation information
    Vendor Status Phase 1 Feedback Attestations Comments (Phase 1) Phase 2 feedback Comments (Phase 2)
    Avast PARTICIPANT Completed Completed None Completed None
    Avira PARTICIPANT Completed Completed None Completed None
    BitDefender PARTICIPANT Completed Completed None None None
    ESET PARTICIPANT Completed Completed None None None
    F-Secure PARTICIPANT Completed Completed None None None
    Kaspersky PARTICIPANT None None (waived) None Completed None
    Microsoft INCLUDED None None None None None
    McAfee INCLUDED None None None None None
    Symantec PARTICIPANT Completed (Waived) Submitted Completed None
    Notable Commentary

    Symantec (Phase 1 – responses from tester in blue):

    3.6.a: Microsoft Edge is not the most widely used browser in the windows OS. Moreover, it is also the most restrictive browser, resulting in a number of Symantec’s protection technologies being unavailable. Based on market share, Chrome would be the most appropriate choice.
    We got the same feedback from multiple vendors. We can’t change this in Q4, but we will experiment with this in Q1.

    3.7: The tester should ensure that malicious actions occurred on the system and refrain from simply checking if the application “ran” as an indicator of protection.
    3.8: Same complaint at 3.7. The tester is advised to verify that malicious actions occurred.

    We are checking for malicious actions. Methodology/test plan will be updated in Q1.

    4.0.7: The tester should clarify if the “Original site” is the one hosting the malware. If so, how would it be ensured that the site is serving a deterministic malicious program and there is no change in content over the period of testing?
    We already use a replay proxy to prevent such issues. Once a content is cached by our replay proxy, the same content is given back during the test. Methodology/test plan will be updated in Q1 to reflect this.

    5.6: Use of internal servers for FP testing is not a real-world scenario. There is also no mention of the scale/impact of a FP in the test plan.
    We will discuss internal server issue internally.
    There is no scale/impact of the FP test, as there is no “one, final ranking” where ITW, performance and FP results are combined together.

    6: In the performance section – the tester should clarify number of runs used and the methodology used to address variations between those runs.
    Agree. Methodology/test plan will be updated in Q1

  • Test Title: “MRG Effitas Q4 2018 360 Degree Assessment and Certification”
    AMTSO Test ID: AMTSO-LS1-TP005

    Test Plan

    Notification date: 2018-10-02
    Notification method: Publicly posted test plan, Contact List notice

    Commencement date: 2018-10-10

    Phase 1 Commentary dates: 2018-11-16 – 2018-11-27
    Phase 2 Commentary dates: 2019-02-11 – 2019-02-18

    Publication date: 2019-02-10

    Published test Report

  • MRG Effitas “Q4 2018 360 Degree Assessment and Certification” (ID AMTSO-LS1-TP005) covered the following products:

    Avast Business Antivirus
    Avira Antivirus Pro – Business edition
    BitDefender Gravityzone Advanced Business Security – cloud
    ESET Endpoint Security
    F-Secure Protection Service for Business
    Microsoft Windows Defender
    McAfee Endpoint Security
    Symantec Endpoint Protection

  • Phase 1 Commentary dates: 2018-11-16 – 2018-11-27
    Phase 2 Commentary dates: 2019-02-11 – 2019-02-18

    Commentary received:

  • Compliance Status: Confirmed compliant with AMTSO Standard v1.1

    Compliance report