Test Lab: MRG Effitas Logo
Test Title: MRG Effitas Q1 2018 360 Degree Assessment and Certification
AMTSO Test ID: AMTSO-PP1-TP202

  • Statement from Test Lab:

    March 2018 – 360 Degree Assessment and Certification

    A first-of-its-kind test that covers all angles, the MRG Effitas 360 Degree Protection Test targets the key threats faced by internet users. In each test case we employ the full spectrum of Early Life Malware. We use a Time-To-Detect metric to measure how long it takes each application to detect and neutralize missed threats.

    Participants: 10/19

    Compliance Status: Partial Compliance
    Note: Some issues were noted in the compliance check; as a complete list of test subjects was not available at the start of the test, proper notification, access to Participant status and configuration audit, and Commentary rights were not provided to all test subjects. Operationally, AMTSO will prevent this situation in future tests and ensure all test subjects receive full notification and per-product requests for Commentary.

    Publication Date: May 17, 2018

    Published Test Report (PDF)

    Participation information
    Test Subject Vendor Status Phase 1 Feedback Attestations Comments (Phase 1) Phase 2 feedback Comments (Phase 2)
    Avast PARTICIPANT None (Waived) None None None
    AVG PARTICIPANT None (Waived) None None None
    Avira PARTICIPANT Completed Completed None None None
    BitDefender PARTICIPANT Completed Completed None NO ISSUES None
    ESET PARTICIPANT Completed Completed None NO ISSUES None
    F-Secure PARTICIPANT Completed Completed SUBMITTED SUBMITTED SUBMITTED
    GData INCLUDED None None None None None
    Kaspersky PARTICIPANT Completed Completed None None None
    Malwarebytes INCLUDED None None None None None
    McAfee INCLUDED None None None None None
    Microsoft PARTICIPANT None (Waived) None None None
    Panda INCLUDED None None None None None
    SurfRight INCLUDED None None None None None
    Symantec INCLUDED None None None None None
    ThreatTrack INCLUDED None None None None None
    Trend Micro INCLUDED None None None None None
    Watchdog INCLUDED None None None SUBMITTED None
    Webroot PARTICIPANT None (Waived) None None None
    Zemana PARTICIPANT None (Waived) None NO ISSUES SUBMITTED
    Notable Commentary:

    F-SECURE (Phase 1):
    “The initial invitation to the test and possible “Voluntary Participant” status we as a company was unknown to me at least until this questionnaire was presented to us. This might be the case with other organizations as well; I propose that whenever such a test is conducted the participant status is clearly outlined when the materials regarding the test itself are being delivered.

    (This would serve two purposes — all the relevant people from the tests perspective would get the information, and it would still allow correction to be made if not everybody agrees with the status.)

    I left the “I confirm that the Tester established a process allowing me to adopt Voluntary Participant status in this Test” checked out as the participant status question wasn’t addressed to me personally at the time it was asked. After discussing with it, it became clear that the decision was presented to us (read: us as in F-Secure Oyj.)”

    ZEMANA (Phase 2):
    “Keep up the good work guys!”

    F-SECURE (Phase 2):
    “I didn’t receive any invitations for a configurations check.“

  • Notification date: January 24, 2018
    Test plan shared on notification

    Test Plan

    Commencement date: Feb 12-Mar 30, 2018
    Phase 1 Commentary Completion Date: February 26, 2018
    Completion date (expected): May 7, 2018
    Phase 2 Commentary Completion Date: May 24, 2018
    Publication date: May 17, 2018
    Compliance report date: June 7, 2018

    Published Test Report (PDF)

  • avast! Internet Security 18.3.2333
    AVG Internet Security 18.3.3051
    Avira Internet Security 15.0.34.27
    BitDefender Internet Security 2018 22.0.20.260
    ESET Internet Security 11.0.159.9
    F-Secure Business Computer Protection 18.4
    GData Internet Security 25.4.0.2
    Kaspersky Internet Security 2018 18.0.0.405 (g)
    Malwarebytes AntiMalware 3.4.4.2398
    McAfee Total Protection 16.0 R10
    Microsoft Windows Defender with SmartScreen 4.12.16299.15
    Panda Internet Security 17.0.1
    SurfRight HitmanPro 3.8.0 Build 292
    Symantec Norton Security 22.14.0.54
    ThreatTrack Vipre Advanced Security 10.1.4.33
    Trend Micro Maximum Security 12.0.1226
    Watchdog Anti-Malware 2.74.186.426
    Webroot SecureAnywhere 9.0.19.43
    Zemana AntiMalware 2.74.2.150

  • Phase 1 Commentary Completion Date: January 31, 2018

    Phase 2 Commentary Completion Date: May 24, 2018

    • Avast

      Vendor Status: PARTICIPANT

      Vendor Attestation Status: NO ATTESTATIONS PROVIDED (Tester waived attestation requirements)

      Commentary (Phase 1): NO COMMENTARY SUBMITTED

      Vendor Confirmation Status: NO RESPONSE

      Commentary (Phase 2): NO COMMENTARY SUBMITTED

    • AVG

      Vendor Status: PARTICIPANT

      Vendor Attestation Status: NO ATTESTATIONS PROVIDED (Tester waived attestation requirements)

      Commentary (Phase 1): NO COMMENTARY SUBMITTED

      Vendor Confirmation Status: NO RESPONSE

      Commentary (Phase 2): NO COMMENTARY SUBMITTED

    • Avira

      Vendor Status: PARTICIPANT

      Vendor Attestation Status: ALL ATTESTATIONS PROVIDED

      Commentary (Phase 1): NO COMMENTARY SUBMITTED

      Vendor Confirmation Status: NO RESPONSE

      Commentary (Phase 2): NO COMMENTARY SUBMITTED

    • Bitdefender

      Vendor Status: PARTICIPANT

      Vendor Attestation Status: ALL ATTESTATIONS PROVIDED

      Commentary (Phase 1): NO COMMENTARY SUBMITTED

      Vendor Confirmation Status: ALL ITEMS MARKED AS SATISFACTORY

      Vendor Confirmation Status: NO RESPONSE

      Commentary (Phase 2): NO COMMENTARY SUBMITTED

    • ESET

      Vendor Status: PARTICIPANT

      Vendor Attestation Status: ALL ATTESTATIONS PROVIDED

      Commentary (Phase 1): NO COMMENTARY SUBMITTED

      Vendor Confirmation Status: ALL ITEMS MARKED AS SATISFACTORY

      Commentary (Phase 2): NO COMMENTARY SUBMITTED

    • F-Secure

      Vendor Status: PARTICIPANT

      Vendor Attestation Status: ALL ATTESTATIONS PROVIDED

      Vendor Confirmation Status: SOME ITEMS MARKED AS LESS THAN SATISFACTORY

      Item “the Tester established a process allowing me to adopt Voluntary Participant status in this Test” – marked as NOT CONFIRMED.

      Commentary (Phase 1): COMMENTARY SUBMITTED as follows:

      “The initial invitation to the test and possible “Voluntary Participant” status we as a company was unknown to me at least until this questionnaire was presented to us. This might be the case with other organizations as well; I propose that whenever such a test is conducted the participant status is clearly outlined when the materials regarding the test itself are being delivered.

      (This would serve two purposes — all the relevant people from the tests perspective would get the information, and it would still allow correction to be made if not everybody agrees with the status.)

      I left the “I confirm that the Tester established a process allowing me to adopt Voluntary Participant status in this Test” checked out as the participant status question wasn’t addressed to me personally at the time it was asked. After discussing with it, it became clear that the decision was presented to us (read: us as in F-Secure Oyj.)”

      Vendor Confirmation Status: SOME ITEMS MARKED AS LESS THAN SATISFACTORY

      Item: “Configuration checks” – marked as “Configuration audit not offered”

      Commentary (Phase 2): COMMENTARY SUBMITTED as follows:

      “I didn’t receive any invitations for a configurations check.“

    • GData

      Vendor Status: INCLUDED

      Vendor Attestation Status: NO ATTESTATIONS PROVIDED

      Commentary (Phase 1): NO COMMENTARY SUBMITTED

      Vendor Confirmation Status: NO RESPONSE

      Commentary (Phase 2): NO COMMENTARY SUBMITTED

    • Kaspersky Labs

      Vendor Status: PARTICIPANT

      Vendor Attestation Status: ALL ATTESTATIONS PROVIDED

      Commentary (Phase 1): NO COMMENTARY SUBMITTED

      Vendor Confirmation Status: NO RESPONSE

      Commentary (Phase 2): NO COMMENTARY SUBMITTED

    • Malwarebytes

      Vendor Status: INCLUDED

      Vendor Attestation Status: NO ATTESTATIONS PROVIDED

      Commentary (Phase 1): NO COMMENTARY SUBMITTED

      Vendor Confirmation Status: NO RESPONSE

      Commentary (Phase 2): NO COMMENTARY SUBMITTED

    • McAfee

      Vendor Status: INCLUDED

      Vendor Attestation Status: NO ATTESTATIONS PROVIDED

      Commentary (Phase 1): NO COMMENTARY SUBMITTED

      Vendor Confirmation Status: NO RESPONSE

      Commentary (Phase 2): NO COMMENTARY SUBMITTED

    • Microsoft

      Vendor Status: PARTICIPANT

      Vendor Attestation Status: NO ATTESTATIONS PROVIDED (Tester waived attestation requirements)

      Commentary (Phase 1): NO COMMENTARY SUBMITTED

      Vendor Confirmation Status: NO RESPONSE

      Commentary (Phase 2): NO COMMENTARY SUBMITTED

    • Panda

      Vendor Status: INCLUDED

      Vendor Attestation Status: NO ATTESTATIONS PROVIDED

      Commentary (Phase 1): NO COMMENTARY SUBMITTED

      Vendor Confirmation Status: NO RESPONSE

      Commentary (Phase 2): NO COMMENTARY SUBMITTED

    • Surfright

      Vendor Status: INCLUDED

      Vendor Attestation Status: NO ATTESTATIONS PROVIDED

      Commentary (Phase 1): NO COMMENTARY SUBMITTED

      Vendor Confirmation Status: NO RESPONSE

      Commentary (Phase 2): NO COMMENTARY SUBMITTED

    • Symantec

      Vendor Status: INCLUDED

      Vendor Attestation Status: NO ATTESTATIONS PROVIDED

      Commentary (Phase 1): NO COMMENTARY SUBMITTED

      Vendor Confirmation Status: NO RESPONSE

      Commentary (Phase 2): NO COMMENTARY SUBMITTED

    • ThreatTrack

      Vendor Status: INCLUDED

      Vendor Attestation Status: NO ATTESTATIONS PROVIDED

      Commentary (Phase 1): NO COMMENTARY SUBMITTED

      Vendor Confirmation Status: NO RESPONSE

      Commentary (Phase 2): NO COMMENTARY SUBMITTED

    • Trend Micro

      Vendor Status: INCLUDED

      Vendor Attestation Status: NO ATTESTATIONS PROVIDED

      Commentary (Phase 1): NO COMMENTARY SUBMITTED

      Vendor Confirmation Status: NO RESPONSE

      Commentary (Phase 2): NO COMMENTARY SUBMITTED

    • Watchdog

      Vendor Status: INCLUDED

      Vendor Attestation Status: NO ATTESTATIONS PROVIDED

      Commentary (Phase 1): NO COMMENTARY SUBMITTED

      Vendor Confirmation Status: SOME ITEMS MARKED AS LESS THAN SATISFACTORY

      Configuration checks: I was not offered an opportunity to audit the configuration of my product
      Adherence to the test plan: I confirm that, to the best of my knowledge, the tester followed the published test plan throughout
      Communications: I confirm that all communications between the tester and my company were handled in a timely, relevant and fair manner
      Feedback and disputes: I was not given an opportunity to review and dispute the test data

      Commentary (Phase 2): NO COMMENTARY SUBMITTED

    • Webroot

      Vendor Status: PARTICIPANT

      Vendor Attestation Status: NO ATTESTATIONS PROVIDED (Tester waived attestation requirements)

      Commentary (Phase 1): NO COMMENTARY SUBMITTED

      Vendor Confirmation Status: PENDING

      Commentary (Phase 2): PENDING

    • Zemana

      Vendor Status: PARTICIPANT

      Vendor Attestation Status: NO ATTESTATIONS PROVIDED (Tester waived attestation requirements)

      Commentary (Phase 1): NO COMMENTARY SUBMITTED

      Vendor Confirmation Status: ALL ITEMS MARKED AS SATISFACTORY

      Commentary (Phase 2): COMMENTARY SUBMITTED as follows:

      Other comments on the execution of this test: “Keep up the good work guys!“

  • Compliance Status: Partial Compliance with Draft Standard v6.1 – 2018-06-05

    Note: Some issues were noted in the compliance check; as a complete list of test subjects was not available at the start of the test, proper notification, access to Participant status and configuration audit, and Commentary rights were not provided to all test subjects. Operationally, AMTSO will prevent this situation in future tests and ensure all test subjects receive full notification and per-product requests for Commentary.

    Compliance Report