SREcon17 Americas Program

Hillary for America was the organization created for Hillary Clinton’s 2016 presidential campaign. By election day, the campaign employed 82 software developers. A team of four SREs and two Security Engineers helped protect and scale close to a hundred backend services against a constant stream of DDoS, looming deadlines and constantly shifting priorities. 

This team helped build an environment to promote building new projects in a week for immovable deadlines, raising billions of dollars from donors, and withstand spiky and semi-unpredictable traffic. We will walk through how they did this, the unusual schedule of a presidential campaign and how that affects a growing tech team.

Over the past two years Shopify built up a strong SRE team focused on the critical systems at the core of the company’s business. However at the same time the company was steadily growing a list of secondary production services, to the point where we had several hundred running applications, most of which had poorly defined ownership and ad-hoc infrastructure. These applications had a patchwork of tools such as build and deployment automation, monitoring, alerting, and load testing, but with very little consistency and lots of gaps. 

Shopify avoided this looming disaster through automation. We built an application that tracked every production service, and linked it to associated owners, source code, and supporting infrastructure. This application, called Services DB, provides a central place where developers can discover and manage their services across various runtime environments. Services DB also provides a platform for measuring progress on infrastructure quality and building out additional tooling to automate manual steps in infrastructure management. Today, Shopify SREs aren’t worried about being woken up in the middle of the night because of failures in poorly maintained applications. Instead, SREs can focus on building automation, and use Services DB to apply it across large groups of services.

Chaos Engineering is described as “the discipline of experimenting on a distributed system in order to build confidence in the system’s capability to withstand turbulent conditions in production.” 

In this presentation I'll go into why we got started with Chaos Engineering, how we built it with a functional programming language, and the road to cultural acceptance. Surprisingly enough, social acceptance proved to be more difficult than actual implementation. There are several different "levels" of chaos that I recommend introducing before unleashing a full-blown chaos solution. I will go over each of the levels, evaluate how the chaos tool has influenced the process, and leave you with a game plan of how you can culturally and technically introduce fault injection in your organization.

Over the past several years Lyft has migrated from a monolith to a sophisticated "service mesh" powered by Envoy, a new high performance open source proxy which aims to make the network transparent to applications. Envoy's out of process architecture allows it to be used alongside any language or runtime. At its core, Envoy is an L4 proxy with a pluggable filter chain model. It also includes a full HTTP stack with a parallel pluggable L7 filter chain. This programming model allows Envoy to be used for a variety of different scenarios including HTTP/2 gRPC proxying, MongoDB filtering and rate limiting, etc. Envoy provides advanced load balancing support including eventually consistent service discovery, circuit breakers, retries, zone aware load balancing, etc. Envoy also has best in class observability using both statistics, logging, and distributed tracing.

In this talk we will discuss why we developed Envoy, focusing primarily on the operational agility that the burgeoning “service mesh” SoA paradigm provides as well as discussing lessons learned along the way.

For more information on Envoy see: https://lyft.github.io/envoy/

Let's face it - the perimeter-based architecture has failed us. Today's attack vectors can easily defeat expensive stateful firewalls and evade IDS systems. Perhaps even worse, perimeters trick people into believing that the network behind it is somehow "safe", despite the fact that chances are overwhelmingly high that at least one device on that network is already compromised.

It is time to consider an alternative approach. Zero Trust is a new security model, one which considers all parts of the network to be equally untrusted. Taking this stance dramatically changes the way we implement security systems. For instance, how useful is a perimeter firewall if the networks on either side are equally untrusted? What is your VPN protecting if the network you're dialing into is untrusted? The Zero Trust architecture is very different indeed. 

In this talk, we'll go over the Zero Trust model itself, why it is so important, what a Zero Trust network looks like, and what components are required in order to actually meet the challenge.

• 4-Minute Docker Deploys—Kat Drobnjakovic, Shopify
• Measuring Reliability through VALET Metrics—Raja Selvaraj, Home Depot
• Lessons Learned from Transforming SEs into SRE at Microsoft Azure—Cezar Alevatto Guimaraes, Microsoft
• SR(securit)E—Tom Schmidt, IBM
• How Three Changes Led to Big Increases in Oncall Health—Dale Neufeld, Shopify