AppEsteem UwS Certification 2019 (series)

Test Lab

AppEsteem

Test Title

AppEsteem UwS Certification 2019 (series)

AMTSO Test ID

AMTSO-LS1-TP006

Platform

Windows, MacOS

Vendor

AVAST, AVG, Avira, Bitdefender, Comodo, Dr Web, Enigmasoft, ESET, G Data, K7 Computing, Kaspersky, MalwareBytes, McAfee, Microsoft, Panda Security, Qihoo, Quick Heal, Sophos, Symantec, Tencent, TG Soft, Trend Micro, VIPRE, Webroot

Publication date

2020-01-14

Statement from Test Lab

“For almost three years, AppEsteem has worked to establish a consensus among AVs about the behaviors that make an app Unwanted Software (UwS) or a Potentially Unwanted Application (PUA). We have codified these behaviors into “ACRs”, or Application Certification Requirements. We publish a freely-available Deceptor® feed, which contains software monetization apps we believe are Unwanted, and a freely-available Certified feed, which contains apps we have certified as clean, and that we believe should be considered neither UwS nor PUA. These feeds are widely used by many parties, including AVs. The objective of this UwS Handling Certification Test is to provide consumers with information on which AV products have effectively implemented our Deceptor and Certified feeds, so they can be assured of effective protection against over-aggressive software monetizers while not getting impeded from working with “clean” software monetizers. Because each AV has control of their own policies, this UwS Handling Certification Test lets AVs describe any deviations they have between their blocking policies and our ACRs. Each AV’s final score reflects their ability to block Deceptors, allow Certifieds, and explain their policy deviations.”

Tested products

VendorProductVendor status
AVASTInternet Securityparticipant
AVGInternet Securityparticipant
AviraInternet Securityparticipant
BitdefenderInternet Securityincluded
ESETInternet Securityincluded
G DataInternet Securityincluded
K7 ComputingTotal Securityparticipant
KasperskyInternet Securityparticipant
MalwareBytesPremiumincluded
McAfeeTotal Protectionincluded
MicrosoftWindows Defenderincluded
Panda SecurityDomeincluded
SophosHome Premiumparticipant
SymantecNorton Security Standardparticipant
TG SoftVirIT eXplorer Liteparticipant
Trend MicroInternet Securityparticipant
ComodoAntivirus (added round 2)included
Dr WebSecurity Space (added round 2)included
Qihoo360 Total Security (added round 2)included
Quick HealInternet Security (added round 2)included
TencentPC Manager (added round 2)included
VIPREAdvanced Security (added round 2)included
WebrootSecureAnywhere (added round 2)included

AMTSO Standard compliance info

Notification issued

2018-12-12

Notification method

Publicly posted test plan, Contact list notification

Test plan

Commencement date

2019-01-07

Participants

9
These Vendors chose to adopt Participant status under the AMTSO Standard, gaining certain guaranteed rights in return for attestations.

“Included” Vendors

14

These Vendors did not chose to adopt Participant status under the AMTSO Standard, but may have engaged with the test lab in other ways.

Commentary dates
CommentaryStart dateEnd date
Phase 1 Commentary2019-01-102019-01-18
Phase 2 Commentary (Round 1 - Jan, Feb)2019-03-072019-03-14
Phase 2 Commentary (Round 2 - Mar, Apr)2019-05-162019-05-23
Phase 2 Commentary (Round 3 - May, Jun) 2019-07-102019-07-18
Phase 2 Commentary (Round 4 - Jul-Sep) 2019-10-142019-10-21
Phase 2 Commentary (Round 5 - Oct-Dec) 2020-01-272020-02-03

Commentary received

VendorCommentary phaseComment
K7 ComputingPhase 2 Jan-Feb

Item “Adherence to the test plan” marked “Acceptable”, comment: “In a certain case one sample which was not outside the described 3-day curation window was used in the test. But on dispute this issue was resolved. In a certain other case one sample for which certification status had expired at the time of the test was used. This was also resolved on dispute.”

Response from tester: “Thanks for this feedback. We automatically curate the samples and these get auto-excluded in our results feeds. However, the dispute user experience was still showing these files. We’ll work on adjusting the dispute process so these auto-excluded samples are more clearly marked as auto-excluded.”

Item “Feedback and Dispute” marked “Problematic”, comment: “Dispute information was automatically submitted without providing us the option of verifying our response beforehand. This issue has been reported to AppEsteem with a request to be able to review our response before submitting.”

Response from tester: “Thanks for this feedback. We’ll find a way to make it more clear that this data is auto-saved.”

Item “Execution of Test” comment: “We would request ongoing stats on the absolute numbers and percentages of detected/missed Deceptors, detected Certified apps. This would allow us to understand how we’re progressing in real time. Thank you.”

Response from tester: “Thanks for this feedback. This capability exists in the user experience, and we have recently added a results feed for each AV that can be checked via automation for updates.”

SophosPhase 2 Jan-Feb

Item “Feedback and Dispute” marked “Problematic”, comment: “The process is functional but it’s still a little rough around the edges. If possible it would be nice to either have individual videos of each test case, or some sort of tagging/labeling/skipping to the proper time in the video rather than having to manually scrub through an hour of footage to find it. It would also be nice to receive email notifications for any FN/FPs or at least a monthly summary rollup email since testing seems to be on random days with no notification at all of any problems.”

Response from tester: “Thanks for this feedback. We have added the timestamp for each sample underneath the video to help you find these. We’ve also added a results feed for the AVs that can be checked via automation for updates.”

AMTSO Standard compliance status

Confirmed compliant with AMTSO Standard v1.1Round 1 (Jan, Feb)
Confirmed compliant with AMTSO Standard v1.1Round 2 (Mar, Apr)
Confirmed compliant with AMTSO Standard v1.1Round 3 (May, Jun)
Confirmed compliant with AMTSO Standard v1.1Round 4 (Jul-Sep)
Confirmed compliant with AMTSO Standard v1.1Round 5 (Oct-Dec)