How to Build Realistic Machine Learning Systems for Security?

Given the existence of adversarial attacks and fairness biases, a question might arise if machine learning is useful for security at all. In this talk, we will discuss how to build robust machine learning systems to defend against real-world attacks. We focus on building machine learning-based malware detectors. We address the necessity of considering RoC curves where the FP rates need to lie well below 1%. Achieving this in the presence of a polluted ground truth set where 10–30% of data is unlabeled and 2–5% of labels are incorrect is a true challenge. When a dynamic model is built, testing it against a repository of malware is impossible, since most malware is ephemeral and may no longer exhibit the malicious property. Finally, we discuss how to model realistic adversaries for adversarial attacks and defenses.