Mandating the Forbidden: Balancing Privacy and Security in Production Abuse Systems

Production-scale abuse systems at major platforms are increasingly the subject of both regulatory mandates and privacy restrictions. Those systems often require algorithmic inspection of private user data, retention policy exceptions to longitudinally track major abuse campaigns, and large numbers of abuse raters with privileged access to production data. We discuss policy exceptions, corpus reduction, false positive minimization, and rater auditing and suggest privacy best practices for these critical but high-risk production systems.