SREcon19 Americas Conference Program

After over a decade of hosting itself in the data center, Etsy.com moved to the Google Cloud Platform (GCP) in 2018. In this talk, I'll go over:

• why the company decided to make the transition
• our architectural approaches to migrating a large monolith, and the difficulties we faced gaining confidence in them
• the assumptions we never knew about and had to fix: in the application code, the infrastructure tooling, and our processes
• cutting over to GCP, safely
• things we learnt running there for the last 9+ months

We'll discuss what challenges does one face while building Service Discovery at scale of millions of processes, tens of millions of clients, and tens of thousands of state changes per second.

A lot of SREs are (or will soon be) responsible for Kubernetes clusters. But what exactly makes up Kubernetes? This talk will dive into the services and systems that make a cluster work, how they interact, and what can go wrong. Kubernetes will no longer be a black box, but a system that can be debugged, reconfigured, and improved to suit every administrators' needs.

Firefighting a broken system is time-sensitive and stressful but becomes even more challenging as teams and systems evolve. As an on-call engineer, scaling processes among humans is an important problem to solve. How do we ramp up new engineers effectively? How can we bring existing on-call engineers up to speed? In this workshop we'll share common myths among new on-call engineers and the Do's and Don'ts of on-call onboarding, as well as run through hands-on activities you can take back to work and apply directly to your own on-call processes.

How many awful meetings have you been to in your life, where people are talking forever and saying nothing, or where people are talking at cross purposes and not listening, or where they're saying things that make everyone feel bad? Have you been in retrospectives like that? (Did it make you never want to attend a retrospective again?)

Let's do better! Come learn practical techniques for facilitating pleasant, productive, welcoming retrospectives (which will improve any meeting you need to run). We will talk about the structure of welcoming language and discuss when it's necessary to interrupt someone. We'll examine what it means for language to include blame and how to reframe blaming conversations. We'll practice the mental work of understanding things that seem contrafactual but are actually just confusing (especially helpful for discussing complex systems). When you leave, you'll be ready to make any meeting or retrospective you're in more comfortable and effective, as a leader or an attendee.

In this talk, the author will share their experience starting new teams, splitting and moving them from both technical and non-technical standpoints. This is ideal for new leaders in charge of SRE wondering when it's time to grow beyond a single team and how to. This is also very valuable for SREs who are interested to know what happens behind the scenes, how to influence such changes and how they can help while avoiding burnout.

The best response to a system outage is not "What did you do?", but "What did we learn?" This session will walk through three system-wide outages at Google, at Stitch Fix, and at WeWork—their incidents, aftermaths, and recoveries. In all cases, many things went right and a few went wrong; also in all cases, because of blameless cultures, we buckled down, learned a lot, and made substantial improvements in the systems for the future. Looking back with the perspective of 20-20 hindsight, all of these incidents were seminal events that changed the focus and trajectory of engineering at each organization. You will leave with a set of actionable suggestions in dealing with customers, engineering teams, and upper management. You will also enjoy a few war stories from the trenches, none of which has been previously told fully in public.

At Google, one of the primary objectives of SRE teams is sublinear scaling: the size and number of SRE teams should grow more slowly than the number of supported services. This talk will describe how one team has implemented this principle. Over the last 3 years, we have increased our portfolio by more than 200% (from 187 to 431 supported services) without additional staffing, and we plan for continued growth up to 1000 services. We will review the extensive automation infrastructure that we have in place, describe ongoing projects (including automated incident handling), and discuss the changes we've made in how we approach SRE - moving away from service-specific production readiness reviews towards automated policy verification and service-agnostic consulting. Audience members will hear about a vision for the long-term role of SRE in large organizations, where sublinear scaling requires not just increasing automation but a cultural shift from providing service-specific expertise to mostly service-independent consulting.

With the popularity of "SRE" as a job role, people have become aware that not all such roles are entirely equivalent. There's been a slack channel on the USENIX-SREcon workspace (https://usenix.org/srecon/slack #sre_between_companies) where people have started to explore these distinctions.

This session will be an opportunity to crowd-source more information. It will be a moderated, audience driven session. Come and tell us what SRE means at your company!

In a large scale environment where engineers are empowered to independently deliver an application from concept to working production system, and in public cloud providers that allow access to do almost anything, there is a unique challenge of implementing and maintaining controls that align with tight banking regulations. I will discuss how we've used a combination of open source tools and our custom automation to solve various challenges such as:

• Limiting public access
• Staying ahead of account resource limits
• Enforcing resource ownership
• Cost control
• Security patching
• Account-impacting mistakes

Artificial intelligence is all around us, from the digitals assistants in our microwaves to the apps we rely on every day. Many of these systems build on APIs and services that use machine learning to provide key features. This talk will describe techniques for building predictable, reliable ML-based services as well as ways to sustain these services through social and technical change. We discuss challenges unique to the reliability of these systems and relate our experiences with ML in our production systems to illustrate our techniques.