SREcon22 Americas Conference Program

Establishing the Return-on-Investment (ROI) for Availability work/projects/investments is really hard. There are some strategies for doing it, which I'll explain in the meat of the presentation. If we don't change the course of common discourse regarding ROI, we're going to miss huge opportunities to invest in SRE, or worse, end up in a regulatory hell that sucks all the fun out of it and produces worse outcomes.

After applying at over 40 companies and participating in more than 110 interviews, I found my role. In this talk, I want to discuss how we can make the interviewing experience better for all parties involved (interviewers, interviewees, hiring managers and recruiters). The session will cover everything from application to offer acceptance and discuss actions we can all take to make SRE hiring better.

This talk presents research collected from the VOID—a new open database of public incident reports. Containing nearly 2,000 reports for 660 organizations, the database allows for more structured review and research about software-related incident reporting. Key results from our research challenge standard industry practices for incident response and analysis, like tracking Mean Time To Resolve (MMTR) and using Root Cause Analysis (RCA) methodology. In particular, we demonstrate how unreliable MTTR can be, and how RCA can lead to environments where people are less likely to admit mistakes and speak up about things that could lead to future incidents. We propose alternate metrics (SLOs and cost of coordination data), practices (Near Miss analysis), and mindsets (humans are the solution, not the problem) to help organizations better learn from their incidents, and make their systems safer and more resilient.

COVID-19 impacted education from kindergarten to college in unforeseen ways almost overnight. The world had to adapt to an online-only mode of education. What happened behind the scenes to make that happen is unlike any typical growth story for a large product or service. This talk will describe the problems, solutions, and design patterns that Microsoft Teams leveraged to be successful at re-enabling education for millions of remote students. We will focus on education-specific scenarios ranging from seasonal tenant onboarding to optimizing for school hardware to changing customer expectations.

Ten years ago, Boeing faced a difficult choice. The Airbus A320neo was racking up orders faster than any plane in history because of its fuel efficiency improvements, and Boeing needed to compete. Should they design a new plane from scratch or just update the tried-and-true 737 with new engines?

The 737 MAX entered service seven years later as the result of that and hundreds of other choices along the way. Let's look at some of those choices in context to understand how the 737 MAX went so very wrong. We'll learn a thing or two along the way about making better decisions ourselves and as teams.

In this talk, you will learn how a sub-second latency query engine inspired by Facebook's Scuba was extended from running in RAM only, to querying the most recent data that could still fit on local SSD, to querying months of data at a time using cloud storage and serverless functions. We'll describe the pitfalls of managing lambdas at scale, including impatience, maximum concurrency, runtime and architecture configuration/experimentation, and the price/performance of renting 20,000 parallel workers.

This talk presents an exciting analytical method that is successfully delivering high fidelity insights useful in analyzing and diagnosing distributed systems. It has been used in production in a variety of complex services at scale (up to 1.4T events/day), where traditional methods have failed, with good results. We will sketch out the problem domain in detail, present the statistical methods used, as well as the intuition behind the approach.

Attendees will gain an alternative lens through which they can analyze performance, as well as an understanding of pitfalls.

What are good alerts? What are bad ones?

The difference is important for reliability. But how do you measure it? What kind of trade-offs are possible?

A model of alert quality will be presented, including parameters like cost and accuracy.

The DevOps Research and Assessment group, or DORA, has conducted broad research on engineering teams’ use of DevOps for nearly a decade. Meanwhile, Site Reliability Engineering (SRE) has emerged as a methodology with similar values and goals to DevOps. How do these movements compare? In 2021, for the first time, DORA studied the use of SRE across technology teams, to evaluate its adoption and effectiveness. We found that SRE practices are widespread, with a majority of teams surveyed employing these techniques to some extent. We also found that SRE works: higher adoption of SRE practices predicts better results across the range of DevOps success metrics. In this talk, we’ll explore the relationship between DevOps and SRE and how even elite software delivery teams can benefit through the continuous modernization of technical operations.

This session will be an inspirational, structured talk where I converge two perspectives on operational debt into a single model based on process gaps and risk. I define operational debt as, “work required to fix process gaps that present a risk to business operations.” This talk is intended to inspire senior SREs and SRE managers to think more holistically about process problems and related automation opportunities. Weaving risk into the approach provides a mechanism to set priority on a risk/reward basis. Creating a backlog of operational debt is also helpful when collaborating with other engineering managers. We will also re-examine Fowler’s quadrants in the operational debt context, compare and contrast operational debt with other forms of debt (financial and technical), and examine how much debt is also toil.