Take Risks... But Don’t Be Stupid

The devops culture values taking risks and learning from failure. Of course, there are reasonable risks... and then there is sheer stupidity. A pre-production environment that supports experimentation and exploration can enable responsible risk-taking. As a system grows, however, it is not feasible simply to deploy two of everything. Building an effective pre-production environment becomes an artful exercise in systems-building in its own right. Our production systems runs on several hundred instances; our pre-production environment uses just a few dozen. I will describe the techniques and trade-offs we use to run a powerful and effective testbed with minimal resources. You can use these ideas in your own systems.