SREcon20 Americas Conference Program

In the aftermath of a serious outage leadership aims to improve performance and avoid future incidents. With so much data to analyze it's difficult to know where to direct attention. How do we know we're getting better? Focusing solely on shallow one-dimensional measures of progress like MTTR, incident count, and severity obscures the deeper lessons. Holding teams to performance metrics based on things they can't control can be demoralizing. Incidents subtly influence organizations through system changes, new designs, budgets, policies, procedures, and hiring. Thorough incident analysis uncovers these unseen influences and their contributions to safety. Incident analysis produces artifacts such as interview transcripts, annotated timelines, contributing factors, and themes. This enables meta-analyses across incidents uncovering previously unseen opportunities. By providing leaders with richer data, it'll unlock insights into reliability, organizational learning, and opportunities for strategic investments. This fosters deeper trust between leaders and practitioners and yields healthier happier teams.

Noisy pagers can lead to pager fatigue, but noisiness is a subjective idea: every team will have their own answer as to how many alerts lead to a noisy pager. There's an assumption that more must mean worse. That led me to wonder: does the number of pages correlate to on-call satisfaction? In this talk, we will look at the results of the study that confirms that a higher number of pages doesn't correlate with on-call satisfaction. The sense of agency to improve the on-call experience and drive the change in a collaborative manner had a stronger impact on team morale than just a number of pages. We will look at different aspects of team culture that have a high impact on engineers' on-call satisfaction and discuss behavioral patterns that contribute to a positive on-call experience.

Jupyter is commonly thought of as a "data science tool". But the same features that make it appealing to data scientists make it appealing for Site Reliability Engineering: dynamic exploration and ability to share results. The talk will set up an "incident" where a cache slowdown is causing site problems and will show how we can use Jupyter to triage and remediate the problem. I'll also cover post-incident best practices: how to make sure that what has been done is properly documented and ready for the incident retrospective.

Chaos Engineering is steadily transforming from a gimmick to a serious, scientific discipline focused on observing and measuring the effects of the failure in systems of all shapes and sizes, in order to verify their behavior experimentally.

Unfortunately the Internet is still full of slogans like "breaking things in production," which—while well-intentioned—can be harmful to the understanding of what Chaos Engineering is really about. In this talk, I'd like to argue that adopting Chaos Engineering can prove to be a very good investment, regardless of the nature of the system in question.

To do that, I'm going to cover three case studies: a single process, a JVM application, and a set of microservices running on Kubernetes.

A low-context SRE culture is one where the info you need to do your job is available, visible, and accessible. Creating a low-context culture improves remote teamwork, new-hire onboarding, enables project hopping, and helps you better handle a page you receive at 2 am.

A high-context SRE culture is one where most knowledge is unspoken, or people depend on history/experience rather than explicit telling. This can be very frustrating for new employees and a disaster in a "remote only"/COVID-19 team.

You'd rather work in a low-context SRE culture right? I'll explain how to lower the context using (1) the power of defaults, (2) "make right easy", and (3) a culture of ubiquitous documentation. I conclude with ways to get people to write documentation (even if they hate writing documentation).

Weather alert!! Vanguard's Chaos and Resilience Engineering team has implemented a home-grown Chaos Engineering platform, "The Climate of Chaos", featuring self-service experiments with meteorology-themed names like Cyclone, Tornado, and Blizzard. In this session, we'll break down the fundamentals of chaos engineering, the value of the practice, and why we decided to build our own platform from scratch, rather than leveraging a vendor product or an open source library. We'll also get into the details of our architecture, our client engagement model, our risk management strategy, and our integrated observability and reporting tooling. By running targeted experiments with our custom tool suite, we have ensured that our systems are ready to weather any storm, no matter how much chaos is in the forecast!

This structured technical talk will be geared towards an audience of engineers, and will be especially applicable to those working in large enterprise environments.

2020 marked the 50th anniversary of the Apollo 13 mission - what was supposed to be a "routine" third landing on the Moon turned into a dramatic odyssey as an explosion nearly destroyed the spacecraft when it was on the way to the Moon.

It was the incredible efforts of the astronauts in space and the engineers on the ground that ensured the safe return of the astronauts.

What were these efforts? While the terms "Site Reliability Engineering", "Observability", "Chaos Engineering", and others did not exist in 1970, NASA's training and preparation of the astronauts and engineers cover many of the principles and practices of modern Site Reliability Engineering.

How did NASA prepare for the mission? How did they work to solve the problems which occurred during the missions? And how did they learn and improve their systems for future missions.

Find out by attending this session.

Kubernetes promised that your pets would become cattle, but cloud environments in practice behave much more like rabbits. They aren't quite a pet, but still high maintenance. They reproduce too quickly, and you know someone in your team cares about each one. How can you securely manage the proliferation of virtual infrastructure across dev, staging, pre-production, and real honest-to-goodness prod environments? How can you ensure product velocity and rapid iteration while ensuring that everything gets the attention and support that it deserves?

The key is understanding how much you and your organization care about each environment. One way to capture this in metrics is by defining and measuring service level objectives (SLOs).

In the early days of the pandemic, the Internet was melting as everyone adjusted to working remotely and adjusting to sudden increases in traffic. Remember the days of routine outages and meltdowns of various top brands? We saw a significant increase in customer activity as we helped small businesses manage their back-office processes from their home offices and mobile phones. During this time, we experienced issues with many of our service providers.

However, instead of melting down, we improved our production resilience, safeguarded against customer-facing incidents, and increased our availability. In this talk, we will discuss how we managed to grow our business and improve our availability by focusing on three things:

• Leading a cross-functional culture change towards extreme ownership of production
• Improved resilience in our payments platform with additional redundancy, self-healing, and retries
• Focusing on third-party management and business processes to scale

Many companies build new-age KVM clouds, only to find out that their applications & workloads do not perform well. In this talk, I'll show the audience how to get the most out of their KVM cloud and how to optimize it for performance: People will understand why performance matters and how to measure it properly. I'll show how to optimize CPU and memory for ultimate performance and how to tune the storage layer for performance. I'll outline what are the main components of an efficient new-age cloud and which network components work best. In addition, I'll show how to select the right hardware to achieve unmatched performance for new-age cloud and applications.