Making "Push On Green" a Reality: Issues & Actions Involved in Maintaining a Production Service

Maintaining a production system is complicated. Systems may consist of many components with separate teams responsible for each one (with multiple development, testing, quality assurance, site reliability, and other engineering teams for each component, each with their own hierarchies, rules, and procedures). Having a system to enforce procedures is a good start, but it is far better to have an automated system to actually perform the updates in a safe and controlled manner. We call this process “Push On Green”. We will discuss some of the many factors that have been (or are actively being) addressed in keeping some of our production systems not only up-and-running, but also updated with as little engineer-involvement and user-visible downtime as possible. In the process, we'll show how you can do this in your environment, too.