Test Lab: MRG Effitas Logo
Test Title: MRG Effitas Q2 2018 360 Degree Assessment and Certification
AMTSO Test ID: AMTSO-LS1-TP001

  • Statement from Test Lab:

    June 2018 – 360 Degree Assessment and Certification

    A first-of-its-kind test that covers all angles, the MRG Effitas 360 Degree Protection Test targets the key threats faced by internet users. In each test case we employ the full spectrum of Early Life Malware. We use a Time-To-Detect metric to measure how long it takes each application to detect and neutralize missed threats.

    Participants: 5/11

    Compliance Status: Confirmed Compliant with AMTSO Standard v1.0

    Publication Date: August 31, 2018

    Published Test Report (PDF)

    Participation information
    Test Subject Vendor Status Phase 1 Feedback Attestations Comments (Phase 1) Phase 2 feedback Comments (Phase 2)
    avast! INCLUDED None None None Completed None
    Avira INCLUDED Submitted None Submitted Completed Submitted
    BitDefender PARTICIPANT Completed Completed None Completed None
    ESET PARTICIPANT Completed Completed None Completed None
    F-Secure PARTICIPANT None (Waived) None Completed None
    Kaspersky PARTICIPANT Completed Completed None Completed Submitted
    Microsoft INCLUDED None None None None None
    McAfee INCLUDED None None None None None
    Symantec INCLUDED Submitted None Submitted Completed Submitted
    Trend Micro INCLUDED None None None None None
    Webroot PARTICIPANT Completed Completed None Completed Submitted
    Notable Commentary:

    AVIRA (Phase 1):
    Avira did not receive any information about Test Subject, notification of the Test Plan and test timing schedule. We did not receive any email which was subjected with “Call for Participant Commentary on MRG-Effitas Q2”. In conclusion, Avira could not made a decision on whether or not to adopt Participant status in the test.

    AMTSO RESPONSE: “AMTSO confirms that notification information was provided by the test lab and duly forwarded to all participants in the AMTSO Contact List system; AMTSO acknowledges and confirms that some the notification messages did not reach some CL participants. This situation has been investigated and addressed. In light of this finding, AMTSO extended the commentary period for this test, including the option to adopt Participant status, to ensure all Test Subject Vendors had time to review the Test Plan and decide on their engagement with the test lab.

    SYMANTEC (Phase 1):
    Symantec did not receive notification of this test or a test plan for this test. There has been no communication pertaining to this test from MRG-Effitas. We are providing feedback, based on a prior version of the test plan that was available on AMTSO website, titled – “MRG Effitas Test Plan for Q1 2018 360 Degree
    Assessment and Certification” from January 22, 2018. We are providing this feedback as a courtesy. We can no longer locate this test plan on the AMTSO member site.

    1) Under section 3, methodology details, 6a – Microsoft Edge is not the most widely used browser in the windows OS. Moreover, it is also the most restrictive browser, resulting in a number of Symantec’s protection technologies being unavailable. Based on market share, Chrome would be the most appropriate choice.

    2) Under section 3, methodology details, 6d – The tester should ensure that malicious actions occurred on the system and refrain from simply checking if the application “ran” as an indicator of protection.

    3) Under section 3, methodology details, 7 – same feedback as #2 above.

    4) Under section 3, methodology details, 8 – same feedback as #2 above. The tester is advised to verify that malicious actions occurred.

    5) Under section 5, how does the tester validate that a sample carries out malicious actions? For e.g. A downloader that does not download any payload should not be a sample used in the test.

    AMTSO RESPONSE: “AMTSO confirms that notification information was provided by the test lab and duly forwarded to all participants in the AMTSO Contact List system; AMTSO acknowledges and confirms that some the notification messages did not reach some CL participants. This situation has been investigated and addressed. In light of this finding, AMTSO extended the commentary period for this test, including the option to adopt Participant status, to ensure all Test Subject Vendors had time to review the Test Plan and decide on their engagement with the test lab.

    See “Test Feedback” tab for full details of feedback submitted by Test Subject Vendors.

  • Notification date: June 6, 2018
    Notification method: Publicly posted Test Plan, Contact List announcement

    Test Plan

    AMTSO Test Identifier: AMTSO-LS1-TP001
    Notification issued: May 16, 2018 (PUBLIC PILOT)
    Notification re-issued: June 6, 2018 (Upgraded to live Standard)
    Notification method: Publicly posted Test Plan
    NOTE: This test was launched in May 2018 under the public pilot; it was re-classified to run under the full Standard after adoption in May 2018. Changes to the Test Plan impact only references to Standard versioning.

    Commencement date: May 23, 2018
    Phase 1 Commentary completion date: June 22, 2018
    Phase 2 Commentary completion date: August 22, 2018
    Publication Date: August 31, 2018

    Published Test Report (PDF)

  • MRG Effitas “Q2 2018 360 Degree Assessment and Certification” covers the following products:

    avast! Internet Security
    Avira Internet Security
    BitDefender Internet Security
    ESET Internet Security (Smart Security)
    F-Secure Computer Protection
    Kaspersky Internet Security
    Microsoft Windows Defender
    McAfee Internet Security
    Symantec Norton Security
    Trend Micro Maximum Security
    Webroot SecureAnywhere

  • Phase 1 Commentary Completion Date: June 22, 2018

    Phase 2 Commentary Completion Date: August 22, 2018

    Avast

    Vendor Status: INCLUDED

    Vendor Attestation Status: NO ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: ALL ITEMS MARKED AS SATISFACTORY

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    Avira

    Vendor Status: INCLUDED

    Vendor Attestation Status: NO ATTESTATIONS PROVIDED

    Vendor Confirmation Status: SOME ITEMS MARKED AS LESS THAN SATISFACTORY

    Item “I received a Test Notification at least five days, but not more than two months, before the Test commenced” – marked as NOT CONFIRMED

    Item “I received a Test Plan which clearly defined the scope, methodology, and schedule for the Test” – marked as NOT CONFIRMED

    Item “the Tester established a process allowing me to adopt Voluntary Participant status in this Test” – marked as NOT CONFIRMED

    Item “all communications between the Tester and my company have been timely, relevant and fair” – marked as NOT CONFIRMED

    Commentary (Phase 1): COMMENTARY SUBMITTED as follows:

    Avira did not receive any information about Test Subject, notification of the Test Plan and test timing schedule. We did not receive any email which was subjected with “Call for Participant Commentary on MRG-Effitas Q2”. In conclusion, Avira could not made a decision on whether or not to adopt Participant status in the test.

    AMTSO RESPONSE: “AMTSO confirms that notification information was provided by the test lab and duly forwarded to all participants in the AMTSO Contact List system; AMTSO acknowledges and confirms that some notification messages did not reach some CL participants. This situation has been investigated and addressed.

    In light of this finding, AMTSO extended the commentary period for this test, including the option to adopt Participant status, to ensure all Test Subject Vendors had time to review the Test Plan and decide on their engagement with the test lab.

    Vendor Confirmation Status: SOME ITEMS MARKED AS LESS THAN SATISFACTORY

    Commentary (Phase 2): COMMENTARY SUBMITTED as follows:

    Avira did not volunteer for the test.

    Bitdefender

    Vendor Status: PARTICIPANT

    Vendor Attestation Status: ALL ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: ALL ITEMS MARKED AS SATISFACTORY

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    ESET

    Vendor Status: PARTICIPANT

    Vendor Attestation Status: ALL ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: ALL ITEMS MARKED AS SATISFACTORY

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    F-Secure

    Vendor Status: PARTICIPANT

    Vendor Attestation Status: Attestation requirements waived by test lab

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: ALL ITEMS MARKED AS SATISFACTORY

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    Kaspersky Labs

    Vendor Status: PARTICIPANT

    Vendor Attestation Status: ALL ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: ALL ITEMS MARKED AS SATISFACTORY

    Commentary (Phase 2): COMMENTARY SUBMITTED as follows:

    Regarding item “Configuration audit process”: “The products are deployed in default settings.

    McAfee

    Vendor Status: INCLUDED

    Vendor Attestation Status: NO ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: NO RESPONSE

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    Microsoft

    Vendor Status: INCLUDED

    Vendor Attestation Status: NO ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: NO RESPONSE

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    Symantec

    Vendor Status: INCLUDED

    Vendor Attestation Status: NO ATTESTATIONS PROVIDED

    Vendor Confirmation Status: SOME ITEMS MARKED AS LESS THAN SATISFACTORY

    Item “I received a Test Notification at least five days, but not more than two months, before the Test commenced” – marked as NOT CONFIRMED

    Item “I received a Test Plan which clearly defined the scope, methodology, and schedule for the Test” – marked as NOT CONFIRMED

    Item “the Tester established a process allowing me to adopt Voluntary Participant status in this Test” – marked as NOT CONFIRMED

    Item “all communications between the Tester and my company have been timely, relevant and fair” – marked as NOT CONFIRMED

    Commentary (Phase 1): COMMENTARY SUBMITTED as follows:

    Symantec did not receive notification of this test or a test plan for this test. There has been no communication pertaining to this test from MRG-Effitas. We are providing feedback, based on a prior version of the test plan that was available on AMTSO website, titled – “MRG Effitas Test Plan for Q1 2018 360 Degree
    Assessment and Certification” from January 22, 2018. We are providing this feedback as a courtesy. We can no longer locate this test plan on the AMTSO member site.

    1) Under section 3, methodology details, 6a – Microsoft Edge is not the most widely used browser in the windows OS. Moreover, it is also the most restrictive browser, resulting in a number of Symantec’s protection technologies being unavailable. Based on market share, Chrome would be the most appropriate choice.

    2) Under section 3, methodology details, 6d – The tester should ensure that malicious actions occurred on the system and refrain from simply checking if the application “ran” as an indicator of protection.

    3) Under section 3, methodology details, 7 – same feedback as #2 above.

    4) Under section 3, methodology details, 8 – same feedback as #2 above. The tester is advised to verify that malicious actions occurred.

    5) Under section 5, how does the tester validate that a sample carries out malicious actions? For e.g. A downloader that does not download any payload should not be a sample used in the test.

    AMTSO RESPONSE: “AMTSO confirms that notification information was provided by the test lab and duly forwarded to all participants in the AMTSO Contact List system; AMTSO acknowledges and confirms that some notification messages did not reach some CL participants. This situation has been investigated and addressed.

    In light of this finding, AMTSO extended the commentary period for this test, including the option to adopt Participant status, to ensure all Test Subject Vendors had time to review the Test Plan and decide on their engagement with the test lab.

    Commentary (Phase 2): COMMENTARY SUBMITTED as follows:

    Item “Communications” – marked as “I have some issues with the way communications between the tester and my company were handled”

    Regarding item “Communications”: “Symantec did not receive the detailed test results. And we did not have enough information to access the results fully.

    Item “Feedback and Disputes” – marked as “I was given an opportunity to review and dispute the test data, but there were issues with the process”

    Trend Micro

    Vendor Status: INCLUDED

    Vendor Attestation Status: NO ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: NO RESPONSE

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    Webroot

    Vendor Status: PARTICIPANT

    Vendor Attestation Status: ALL ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: ALL ITEMS MARKED AS SATISFACTORY

    Commentary (Phase 2): COMMENTARY SUBMITTED as follows:

    Regarding item “Configuration audit process”: “Historically this hasn’t been necessary, default configuration is what is recommended.

  • Compliance Status: Confirmed Compliant with AMTSO Standard v1.0

    Compliance Report (PDF)