Error Budgets and Risks

Striving for Imperfection: Using an error budget to move fast without compromising high reliability

You may assume Site Reliability Engineers aim to build systems that never go down. What that fails to realize is that 100% reliability is almost never the goal. Instead, our task is to trade off reliability against the many other goals we have for our services. SREs want to provide great service to end users and customers, and also have the flexibility to change the systems often and quickly. We want to ensure that the queries and the revenue keep flowing, and do so as efficiently as possible, provisioning as little excess as necessary to deliver good service. Taking an engineering approach to meeting these goals means we need to make these tradeoffs measurable, and this is where error budgets come in. Your error budget is a measure of risk, it is the amount of headroom you have above your SLA. Being smart about how you manage and spend this error budget is one of the best tools that SRE has to meet the various contending goals that services at Internet scale present.

Marc Alvidrez is a Senior Staff Site Reliability Engineer with Google. He joined the company in 2004, and starting as an early SRE he has led a variety of teams responsible for both infrastructure and major user-facing services. These have included the first team responsible for Google File System (GFS), and the teams responsible for Google's Display and AdSense advertising serving systems, Google+ and Google Photos. Prior to Google he held systems engineering roles at Vodafone and Internet startup Topica, where he was the Director of Operations.