Secrets at Scale: Automated Bootstrapping of Secrets & Identity in the Cloud

Almost any web service needs secrets to operate. Whether it’s encryption keys for storing credit card data and personally identifiable information, authentication tokens for talking to third party services, or just a password for connecting to the local database, if your application lives online it probably has a secret. But how do you actually keep those secrets secret? In an ideal world access would be tightly restricted; neither developers, nor continuous integration, nor deployment tools would ever see them. But for applications deployed in the cloud which need to automatically instantiate new instances to match demand and replace unhealthy nodes, this creates an even greater challenge; how can an application be automatically deployed with its secrets if even the deployment tools can’t be allowed to see them?

In this talk I will describe how we have approached this problem at Netflix: an environment supporting thousands of independent microservice applications, all of which need the capability to automatically scale and self-heal. Along the way, I’ll describe how this problem becomes inexorably intertwined with the question of secure, provable, and ephemeral identity, and how we ultimately architected a solution to both problems.