Incident Analysis

Outages and incidents happen. Sh*t breaks, fibers get cut, bugs get pushed to production, teams fail to communicate, and all hell breaks loose. But those who don't learn from mistakes are doomed to repeat them...over and over and over again, with increasing frustration for those on the frontlines fixing the problems and from the users who suffer the impacts.

In an effort to better learn from what happened across all products and services, Google launched an initiative in 2014 to gather data from all outages and incidents that occurred on production systems for trend analysis into system and user impacts, incident timelines, and root causes. The data is then used to drive improvements across systems, processes, and tools to improve the balance between system stability and development velocity. This talk aims to share Google's approach to setting up and running such an analysis program, some preliminary results, and lessons learned.

Sue Lueder joined Google as a Site Reliability Program Manager in 2014 and is on the team responsible for disaster testing and readiness, incident management processes and tools, and incident analysis. Previous to Google, Sue was a technical program manager and a systems, software, and quality engineer in wireless and smart energy industries (OnRamp Wireless, Texas Instruments, Qualcomm). She has a M.S. in Organization Development from Pepperdine University and a B.S in Physics from UCSD.