Content-Oblivious Trust and Safety Techniques: Results from a Survey of Online Service Providers

In pressuring online service providers to better police harmful content on their services, regulators tend to focus on trust and safety techniques, such as automated systems for scanning or filtering content on a service, that depend on the provider's capability to access the contents of users' files and communications at will. I call these techniques content-dependent. The focus on content analysis overlooks the prevalence and utility of what I call content-oblivious techniques: ones that do not rely on guaranteed at-will access to content, such as metadata-based tools and users' reports flagging abuse which the provider did not (or could not) detect on its own.

This talk presents the results of a survey about the trust and safety techniques employed by a group of online service providers that collectively serve billions of users. The survey finds that abuse-reporting features are used by more providers than other techniques such as metadata-based abuse detection or automated systems for scanning content, but that the providers' abuse-reporting tools do not consistently cover the various types of abuse that users may encounter on their services, a gap I recommend they rectify. Finally, despite strong consensus among participating providers that automated content scanning is the most useful means of detecting child sex abuse imagery, they do not consider it to be nearly so useful for other kinds of abuse.

These results indicate that content-dependent techniques are not a silver bullet against abuse. They also indicate that the marginal impact on providers' anti-abuse efforts of end-to-end encryption (which, controversially, stymies providers' ability to access user content at will) can be expected to vary by abuse type. These findings have implications for policy debates over the regulation of online service providers' anti-abuse obligations and their use of end-to-end encryption.