Test Lab: AppEsteem Logo
Test Title: AppEsteem UwS Certification 2019 (series)
AMTSO Test ID: AMTSO-LS1-TP006

  • Statement from Test Lab:

    “For almost three years, AppEsteem has worked to establish a consensus among AVs about the behaviors that make an app Unwanted Software (UwS) or a Potentially Unwanted Application (PUA). We have codified these behaviors into “ACRs”, or Application Certification Requirements. We publish a freely-available Deceptor® feed, which contains software monetization apps we believe are Unwanted, and a freely-available Certified feed, which contains apps we have certified as clean, and that we believe should be considered neither UwS nor PUA. These feeds are widely used by many parties, including AVs.

    The objective of this UwS Handling Certification Test is to provide consumers with information on which AV products have effectively implemented our Deceptor and Certified feeds, so they can be assured of effective protection against over-aggressive software monetizers while not getting impeded from working with “clean” software monetizers.

    Because each AV has control of their own policies, this UwS Handling Certification Test lets AVs describe any deviations they have between their blocking policies and our ACRs. Each AV’s final score reflects their ability to block Deceptors, allow Certifieds, and explain their policy deviations.”

    Test Plan

    Participants: 9
    These Vendors chose to adopt Participant status under the AMTSO Standard, gaining certain guaranteed rights in return for attestations

    “Included” Vendors: 8
    These Vendors did not chose to adopt Participant status under the AMTSO Standard, but may have engaged with the test lab in other ways

    Compliance Status: Confirmed compliant with AMTSO Standard v1.1 – Test components January, February 2019

    TEST SERIES STILL OPEN TO NEW PARTICIPANTS

    Published test Report

    Publication Date: Rolling (new data published monthly)

    Participation information
    Vendor Status Phase 1 Feedback Attestations Comments (Phase 1) Phase 2 feedback Comments (Phase 2)
    Avast PARTICIPANT Completed Completed None Completed None
    AVG PARTICIPANT Completed Completed None Completed None
    Avira PARTICIPANT Completed Completed None Completed None
    Bitdefender INCLUDED None None None None None
    ESET INCLUDED None None None None None
    G_DATA INCLUDED None None None None None
    K7 PARTICIPANT Completed Completed None Completed Submitted
    Kaspersky PARTICIPANT Completed Completed None None None
    MalwareBytes INCLUDED None None None None None
    McAfee INCLUDED None None None None None
    Microsoft INCLUDED None None None None None
    Panda INCLUDED None None None None None
    Sophos PARTICIPANT Completed Completed None Completed Submitted
    Symantec PARTICIPANT Completed Completed None None None
    TGSoft PARTICIPANT Completed Completed None Completed None
    Trend_Micro PARTICIPANT Completed Completed None Completed None
    Webroot PARTICIPANT Completed Completed None Completed None
    Notable Commentary

    K7 Computing (Phase 2 Jan-Feb):

    Item “Adherence to the test plan” marked “Acceptable”, comment:

    In a certain case one sample which was not outside the described 3-day curation window was used in the test. But on dispute this issue was resolved. In a certain other case one sample for which certification status had expired at the time of the test was used. This was also resolved on dispute.

    Response from tester: “Thanks for this feedback. We automatically curate the samples and these get auto-excluded in our results feeds. However, the dispute user experience was still showing these files. We’ll work on adjusting the dispute process so these auto-excluded samples are more clearly marked as auto-excluded.”

    Item “Feedback and Dispute” marked “Problematic”, comment:

    Dispute information was automatically submitted without providing us the option of verifying our response beforehand. This issue has been reported to AppEsteem with a request to be able to review our response before submitting.

    Response from tester: “Thanks for this feedback. We’ll find a way to make it more clear that this data is auto-saved.”

    Item “Execution of Test” comment:

    We would request ongoing stats on the absolute numbers and percentages of detected/missed Deceptors, detected Certified apps. This would allow us to understand how we’re progressing in real time. Thank you.

    Response from tester: “Thanks for this feedback. This capability exists in the user experience, and we have recently added a results feed for each AV that can be checked via automation for updates.”

    Sophos (Phase 2 Jan-Feb):

    Item “Feedback and Dispute” marked “Problematic”, comment:

    The process is functional but it’s still a little rough around the edges. If possible it would be nice to either have individual videos of each test case, or some sort of tagging/labeling/skipping to the proper time in the video rather than having to manually scrub through an hour of footage to find it. It would also be nice to receive email notifications for any FN/FPs or at least a monthly summary rollup email since testing seems to be on random days with no notification at all of any problems.

    Response from tester: “Thanks for this feedback. We have added the timestamp for each sample underneath the video to help you find these. We’ve also added a results feed for the AVs that can be checked via automation for updates.”

  • Test Title: “AppEsteem UwS Certification 2019 (series)”
    AMTSO Test ID: AMTSO-LS1-TP006

    Test Plan

    Notification date: 2018-12-12
    Notification method: Publicly posted test plan, Contact List notice

    Commencement date: 2019-01-07

    Phase 1 Commentary dates: 2019-01-10 – 2019-01-18
    Phase 2 Commentary dates: 2019-03-07 – 2019-03-14 (Jan, Feb components)

    Publication date: 2019-03-14 (Jan, Feb components)

    Published test Report

  • AppEsteem “UwS Certification 2019 (series)” (ID AMTSO-LS1-TP006) covered the following products:
    NOTE: Not all products included in all monthly reports

    Avast Internet Security
    AVG Internet Security
    Avira Antivirus Pro
    Bitdefender Internet Security
    ESET Internet Security
    G DATA INTERNET SECURITY
    K7 Total Security
    Kaspersky Internet Security
    MalwareBytes Premium
    McAfee Total Protection
    Microsoft Windows Defender
    Panda Dome
    Sophos Home Premium
    Symantec Norton Security Standard
    TGSoft VirIT eXplorer Lite
    Trend Micro Internet Security
    Webroot SecureAnywhere

  • Phase 1 Commentary dates: 2019-01-10 – 2019-01-18
    Phase 2 Commentary dates: 2019-03-07 – 2019-03-14 (Jan, Feb components)

    Commentary received:

    Avast

    Vendor Status: PARTICIPANT

    Vendor Attestation Status: ALL ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: ALL ITEMS MARKED AS SATISFACTORY

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    AVG

    Vendor Status: PARTICIPANT

    Vendor Attestation Status: ALL ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: ALL ITEMS MARKED AS SATISFACTORY

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    Avira

    Vendor Status: PARTICIPANT

    Vendor Attestation Status: ALL ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: ALL ITEMS MARKED AS SATISFACTORY

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    Bitdefender

    Vendor Status: INCLUDED

    Vendor Attestation Status: NO ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: NO RESPONSE

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    Eset

    Vendor Status: INCLUDED

    Vendor Attestation Status: NO ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: NO RESPONSE

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    G Data

    Vendor Status: INCLUDED

    Vendor Attestation Status: NO ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: NO RESPONSE

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    K7 Computing

    Vendor Status: PARTICIPANT

    Vendor Attestation Status: ALL ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: SOME ITEMS MARKED AS LESS THAN SATISFACTORY

    Commentary (Phase 2): COMMENTARY SUBMITTED as follows:

    Item “Adherence to the test plan” marked “Acceptable”, comment:

    In a certain case one sample which was not outside the described 3-day curation window was used in the test. But on dispute this issue was resolved. In a certain other case one sample for which certification status had expired at the time of the test was used. This was also resolved on dispute.

    Item “Feedback and Dispute” marked “Problematic”, comment:

    Dispute information was automatically submitted without providing us the option of verifying our response beforehand. This issue has been reported to AppEsteem with a request to be able to review our response before submitting.

    Item “Execution of Test” comment:

    We would request ongoing stats on the absolute numbers and percentages of detected/missed Deceptors, detected Certified apps. This would allow us to understand how we’re progressing in real time. Thank you.

    Kaspersky Lab

    Vendor Status: PARTICIPANT

    Vendor Attestation Status: ALL ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: ALL ITEMS MARKED AS SATISFACTORY

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    MalwareBytes

    Vendor Status: INCLUDED

    Vendor Attestation Status: NO ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: NO RESPONSE

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    McAfee

    Vendor Status: INCLUDED

    Vendor Attestation Status: NO ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: NO RESPONSE

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    Microsoft

    Vendor Status: INCLUDED

    Vendor Attestation Status: NO ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: NO RESPONSE

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    Panda

    Vendor Status: INCLUDED

    Vendor Attestation Status: NO ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: NO RESPONSE

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    Sophos

    Vendor Status: PARTICIPANT

    Vendor Attestation Status: ALL ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: SOME ITEMS MARKED AS LESS THAN SATISFACTORY

    Commentary (Phase 2): COMMENTARY SUBMITTED as follows:

    Item “Feedback and Dispute” marked “Problematic”, comment:

    The process is functional but it’s still a little rough around the edges. If possible it would be nice to either have individual videos of each test case, or some sort of tagging/labeling/skipping to the proper time in the video rather than having to manually scrub through an hour of footage to find it. It would also be nice to receive email notifications for any FN/FPs or at least a monthly summary rollup email since testing seems to be on random days with no notification at all of any problems.

    Symantec

    Vendor Status: PARTICIPANT

    Vendor Attestation Status: ALL ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: NO RESPONSE

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    TG Soft

    Vendor Status: PARTICIPANT

    Vendor Attestation Status: ALL ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: ALL ITEMS MARKED AS SATISFACTORY

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    Trend Micro

    Vendor Status: PARTICIPANT

    Vendor Attestation Status: ALL ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: ALL ITEMS MARKED AS SATISFACTORY

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

    Webroot

    Vendor Status: PARTICIPANT

    Vendor Attestation Status: ALL ATTESTATIONS PROVIDED

    Commentary (Phase 1): NO COMMENTARY SUBMITTED

    Vendor Confirmation Status: ALL ITEMS MARKED AS SATISFACTORY

    Commentary (Phase 2): NO COMMENTARY SUBMITTED

  • Compliance Status: Confirmed compliant with AMTSO Standard v1.1

    Compliance report