The Practical Divide between Adversarial ML Research and Security Practice: A Red Team Perspective

Researchers in adversarial machine learning (ML)—which studies the security of ML in the presence of an attacker—have written more than 2,300 papers in just the last 6 years. However, in a recent survey, almost 90% of corporations and government entities reported that they are not equipped to protect their ML models. The fact remains that there are significant gaps between academic advancements and industry needs and practices. First, aside from full-scale security products, ML models get scant security consideration. Second, most organizations who are concerned with ML security do not know how to begin. Third, the tools and techniques produced in academic research often don't translate to business needs.

To demonstrate these gaps, I will review some sobering lessons learned during a Machine Learning Red Team engagement at Microsoft on systems that rely on ML. They include the reality that "traditional" security measures like access control and permissions remain the most significant element for safeguarding ML systems. Furthermore, especially outside of security applications, awareness about ML vulnerabilities remains low and security posture is nearly zero.