Test Lab: MRG Effitas Logo
Test Title: MRG Effitas Q2 2019 360 Degree Assessment and Certification
AMTSO Test ID: AMTSO-LS1-TP010

  • Statement from Test Lab:

    “A first-of-its-kind test that covers all angles, our pioneering 360 Degree Protection Test targets the key threats faced by internet users. In each test case we employ the full spectrum of Early Life Malware. We use a Time-To-Detect metric to measure how long it takes each application to detect and neutralize missed threats.”

    Test Plan

    Participants: 7
    These Vendors chose to adopt Participant status under the AMTSO Standard, gaining certain guaranteed rights in return for attestations

    “Included” Vendors: 5
    These Vendors did not chose to adopt Participant status under the AMTSO Standard, but may have engaged with the test lab in other ways

    Compliance Status: Confirmed compliant with AMTSO Standard v1.1

    Published test Report

    Publication Date: 2019-08-21

    Participation information
    Vendor Status Phase 1 Feedback Attestations Comments (Phase 1) Phase 2 feedback Comments (Phase 2)
    Avast PARTICIPANT Completed Completed None Completed None
    Avira PARTICIPANT Completed Completed None Completed None
    BitDefender PARTICIPANT Completed Completed None Completed None
    CrowdStrike PARTICIPANT Completed Completed None Completed None
    ESET INCLUDED None None None None None
    F-Secure PARTICIPANT None (Waived) None Completed None
    Kaspersky INCLUDED None None None None None
    Microsoft INCLUDED None None None None None
    McAfee INCLUDED None None None None None
    Sophos PARTICIPANT Completed Completed None Completed None
    Symantec PARTICIPANT Completed Completed Submitted Completed None
    Trend Micro INCLUDED None None None None None
    Notable Commentary

    Symantec (Phase 1):

    Section 4.b: The Window File Explorer has an known issue that incorrectly reports the actual disk utilization where ‘hardlinks’ are used with applications. Norton Security uses hardlinks and Windows File Explorer incorrectly repots the amount of space being used by Norton Security. For the detailed description of Windows File Explorer, please refer to the Microsoft documentation (https://blogs.technet.microsoft.com/mrsnrub/2008/10/25/dont-trust-all-your-eyes-tell-you-hard-links-in-windows-vista/). Our recommendation is to consider using an advanced tool called ‘Disk Usage’ from Microsoft. The tool is available at this link https://docs.microsoft.com/en-us/sysinternals/downloads/du and it reports the correct and accurate disk utilization of Windows applications. We can provide a document that has an example of Windows Explorer is reporting a disk utilization of 9.10 TB where the actual size of C drive is only 753GB.

    Section 4.d: “”Working set”” displayed in the task manager is not a good representation of how much memory is being used by a process. A better reflection would be “”virtual private bytes”” as a measure.

    General performance: c. 3 iterations for evaluating performance is too less. Ideally the each set should consist of at least 10 iterations.

  • Test Plan

    Notification date: 2019-04-09
    Notification method: Publicly posted test plan, Contact List notice

    Commencement date: 2019-04-23

    Phase 1 Commentary dates: 2019-05-09 – 2019-05-17
    Phase 2 Commentary dates: 2019-08-26 – 2019-09-03

    Publication date: 2019-08-21

    Published test Report

  • MRG Effitas “Q2 2019 360 Degree Assessment and Certification” (ID AMTSO-LS1-TP010) covered the following products:

    Avast Business Antivirus
    Avira Antivirus Pro
    BitDefender Endpoint Security
    CrowdStrike Falcon Protect
    ESET Endpoint Security
    F-Secure Computer Protection Premium
    Kaspersky Small Office Security
    Microsoft Windows Defender
    McAfee Endpoint Security
    Sophos Intercept X
    Symantec Endpoint Protection
    Trend Micro Worry-Free Business Security

  • Phase 1 Commentary dates: 2019-05-09 – 2019-05-17
    Phase 2 Commentary dates: 2019-08-26 – 2019-09-03

    Commentary received:

    Avast

    Vendor Status: PARTICIPANT

    Vendor Attestation Status: ALL ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: ALL ITEMS MARKED AS SATISFACTORY

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    Avira

    Vendor Status: PARTICIPANT

    Vendor Attestation Status: ALL ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: ALL ITEMS MARKED AS SATISFACTORY

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    Bitdefender

    Vendor Status: PARTICIPANT

    Vendor Attestation Status: ALL ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: ALL ITEMS MARKED AS SATISFACTORY

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    CrowdStrike

    Vendor Status: PARTICIPANT

    Vendor Attestation Status: ALL ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: ALL ITEMS MARKED AS SATISFACTORY

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    Eset

    Vendor Status: INCLUDED

    Vendor Attestation Status: NO ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: NO RESPONSE

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    F-Secure

    Vendor Status: PARTICIPANT

    Vendor Attestation Status: NO ATTESTATIONS PROVIDED (Waived)

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: ALL ITEMS MARKED AS SATISFACTORY

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    Kaspersky Lab

    Vendor Status: INCLUDED

    Vendor Attestation Status: NO ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: NO RESPONSE

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    McAfee

    Vendor Status: INCLUDED

    Vendor Attestation Status: NO ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: NO RESPONSE

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    Microsoft

    Vendor Status: INCLUDED

    Vendor Attestation Status: NO ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: NO RESPONSE

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    Sophos

    Vendor Status: PARTICIPANT

    Vendor Attestation Status: ALL ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: ALL ITEMS MARKED AS SATISFACTORY

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    Symantec

    Vendor Status: PARTICIPANT

    Vendor Attestation Status: ALL ATTESTATIONS PROVIDED

    Vendor Feedback (Phase 1): Item “Communications” marked “LESS THAN SATISFACTORY

    Commentary (Phase 1): COMMENTARY SUBMITTED as follows:

    Section 4.b: The Window File Explorer has an known issue that incorrectly reports the actual disk utilization where ‘hardlinks’ are used with applications. Norton Security uses hardlinks and Windows File Explorer incorrectly repots the amount of space being used by Norton Security. For the detailed description of Windows File Explorer, please refer to the Microsoft documentation (https://blogs.technet.microsoft.com/mrsnrub/2008/10/25/dont-trust-all-your-eyes-tell-you-hard-links-in-windows-vista/). Our recommendation is to consider using an advanced tool called ‘Disk Usage’ from Microsoft. The tool is available at this link https://docs.microsoft.com/en-us/sysinternals/downloads/du and it reports the correct and accurate disk utilization of Windows applications. We can provide a document that has an example of Windows Explorer is reporting a disk utilization of 9.10 TB where the actual size of C drive is only 753GB.

    Section 4.d: “”Working set”” displayed in the task manager is not a good representation of how much memory is being used by a process. A better reflection would be “”virtual private bytes”” as a measure.

    General performance: c. 3 iterations for evaluating performance is too less. Ideally the each set should consist of at least 10 iterations.

    Vendor Confirmation Status: NO RESPONSE

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    Trend Micro

    Vendor Status: INCLUDED

    Vendor Attestation Status: NO ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: NO RESPONSE

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

  • Compliance Status: Confirmed compliant with AMTSO Standard v1.1

    Compliance report