Today we have invited Scott Jeffreys, AMTSO’s Standards Program Manager and member of the Executive Team, to join us for our next interview segment. Before joining AMTSO, Scott spent 20 years at Reuters where he grew to become the Senior Vice President and Head of Regional Development. He also made a career for himself at CA Technologies, Total Defense Inc., and Honeywell before becoming an Associate Professor at Hofstra University. At Hofstra, Scott teaches many classes within the Computer Science degree curriculum and has become the Graduate Program Director.
Scott, as a program manager at AMTSO you are verifying tests adhering to the AMTSO Standard, could you tell us a bit about what this involves? What are the processes tests go through in this process?
Each Testing Organization that leverages the AMTSO Standard develops a Test Plan which is publicly circulated and posted. The Test Plan provides each test subject vendor with details on the methodology and scope of the testing process. Sufficient transparency is provided in the Test Plan to allow readers to assess that the test will be conducted fairly and that an open dialog for disagreement can take place.
Each test undergoes two rounds of commentary from the test subject vendors. Phase 1 allows for comments to be raised on the Test Plan itself while Phase 2 analyzes how the test was conducted.
Once the Testing Organization publishes their Test Results and AMTSO has collected all of the commentary, a final Compliance Confirmation Report is issued. Here, AMTSO evaluates the test and its execution against fifty checkpoints covering Notifications, Test Plan Content, Participation, Process, Reports, and Attestations. If these checkpoints are satisfied, the test is Confirmed Compliant, the AMTSO term which identifies a test that met the requirements of the Standard.
Why, from your perspective, is the standard important?
The AMTSO Standard provides the framework on which the Tester-Vendor communications are built along with the definitions for what can be examined during the testing cycle. The Standard therefore provides a protocol for transparency and fairness in the testing cycle.
All other activities flow from the Standard itself. Everything from the Test Plan schedules, notification periods, commentary collection, and final Compliance Confirmation Reporting emanate are based on this approach of transparency.
You have played a crucial part at developing the AMTSO Standard, and are involved in the continued evolution and development of the Standard. What are the biggest challenges in this process?
The most obvious challenge is that we must gain consensus from our membership as new elements are added to the Standard. We must make sure that every member’s voice is heard during the update process and we carefully construct language acceptable to all stakeholders. While our current version (Version 1.3) has been in place for about two years, it stands on the shoulders of work done over the last decade.
We also need to keep an ongoing log of requested changes. While this has greatly slowed over the last year as the Standard has become more entrenched, we regularly review change requests as part of the Standards Working Group (SWG) activities.
Third and critically important is education. Our ability to explain the operation of the Standard in clear, concise terms when working with Testers, Vendors, or independent members will always be a work-in-progress.
How has the Standard evolved the past years to embrace technology changes?
We have found that the AMTSO Standard has been able to handle new testing types beyond the classic antimalware environment. Recently, Cloud WAF testing was done under the Standard and we are looking at parallel guidelines to help define testing in the IoT Market.
Our working relationship with NetSecOpen that was announced in 2021 illustrates how AMTSO looks to embrace adjacent testing markets while working to improve our own Standard.
You are working as a Professor of Computer Science and Mathematics at Hofstra University. Do you teach students about fairness in product comparisons and testing? Do you have discussions around these?
After students have been through the introductory programming (Python, C++, Java) and mathematics (discrete structures, calculus) courses, they will come across courses entitled “Computers and Ethics” and “Software Engineering”. In both courses, the ACM Code of Ethics and Professional Conduct is discussed in detail highlighting the characteristics of ethical behavior in Computer Science and Cybersecurity. The Code is designed to inspire and guide the ethical conduct of all computing professionals, including current and aspiring practitioners, instructors, students, influencers, and anyone who uses computing technology in an impactful way.
Therefore, the transition process from programmer to professional is not governed by a licensing body, but rather the training that we instill in our classroom instruction. Providing transparency and fairness as we do with the AMTSO Standard are intrinsic characteristics of ethical behavior including the Code requirements for honesty, trustworthiness, and fairness. The Standard is used as an example where such principles can be applied in the real world.
Are there any lessons from AMTSO that you’ve brought to the classroom or vice versa?
Never underestimate the passion of dedicated Computer Science and Cybersecurity professionals who are looking to do the right thing for their customers and the industry as a whole. While some have speculated that the Cyber-Protection world is a dark market, the classroom has made it possible to show how processes from Ethical Hacking and Network Analysis help make the cyberworld safer.
How do you think the university/academic world can benefit from an entity like AMTSO?
Universities doing research into malware methodologies would greatly benefit from our Real Time Threat List (RTTL) and being part of our Testing Town hall lecture series.
Anything else you’d like to add?
Universities are a wonderful training ground however partnerships with Cybersecurity firms are critical to help drive that training from the theoretical to the practical. Excellent candidates are available, not only at Hofstra University, but other programs throughout the region. While many of the students are quickly scooped up by FinTech, Banking, Medical, and Engineering firms, we also need to guarantee that stream of fresh talent into our Cybersecurity partners within AMTSO.
Thank you so much for your insights and academic perspective Scott!
Statements made by individuals may not reflect the views or opinions of their employers, of other AMTSO members, or of the organization as a whole.