Deploying Changes to Production in the Age of the Microservice

You decoupled your APIs from their implementations and put them behind RPC interfaces. You build and deploy services independently. You code health is impeccable. You put your user data in a persistent, replicated, and consistent store, where it belongs. Your developer velocity has skyrocketed.

Now we have new problems. We’ve got N independent services with M edges of interaction between them. That’s N services that need to be built, tested, and deployed on the infrastructure that expected you to have one service whose mess of entanglement was a secret you had with the compiler.

How do we deploy N binaries with N sources of static configuration and M sources of runtime configuration safely without losing our collective minds? In this talk, I’ll share some of how we grew that aforementioned N from 1 to many in Gmail. Specifically:

• Consistent naming schemas for services, environments -
• Maintaining lightweight, easy-to-change production configuration abstraction layers
• Release early, often
• Canary everything by sharding into more A/B environments than you'd think you’d need 
• Encourage backwards compatibility in all APIs 
• Validate and test all configuration before changing global state

And, of course, some of things we (Gmail) learned by breaking things along the way.