Conference Programme

The principles and constructs of DevOps are pervading the industry and have lit the path for the capability to execute with both speed and quality in balance while managing hockey stick growth. SRE has established itself as the most effective incarnation of DevOps in our industry. Most companies and organizations are nowhere near “SRE goal state.” As this audience knows, building SRE, and applying the devops principles into existing companies and service code bases involves cultural engineering as well as deep tech investments. SRE is a step function change in many dimensions and requires new "genes" to be inserted into an organization's DNA. There is no larger enterprise cloud+consumer company in the world than Microsoft, and we’re on the journey now and investing heavily to shift to SRE. The core of Microsoft was not born in the cloud, but instead in first gen consumer (PC), and then rose to dominate Enterprise. There are a handful of services at Microsoft that were born in the cloud and have scaled massively, with Bing Search being the “granddaddy” with 13 years of experience in the pocket from which we have learned much. Microsoft Azure is the Enterprise Cloud. We are making an enormous investment to run at galactic scale to be the infrastructure for the world’s infrastructure and compete relentlessly in the market with top competitors. In this talk we will compare and contrast the journey within Bing to the current state of execution and operations and how we are taking the lessons from that experience, inspiration from industry, and learnings to date as we build SRE within Azure.

Greg is the Director of the Azure Site Reliability Engineering team in Azure, Microsoft’s cloud infrastructure—a multi-billion-dollar business that is the foundation of the company’s service offerings. Azure is deployed across all geographies and used worldwide by millions of customers. In addition to first party workloads such as Office 365, Bing, and Dynamics, Azure is leveraged by multi-billion-dollar Fortune 500s, small and medium size businesses, as well as startups. The production footprint is enormous—rivaled only by a couple companies on the planet. Azure is uniquely positioned in the Enterprise Cloud space and its service offerings include IaaS, PaaS, and SaaS (infrastructure, platform, and software as a service).

Outside of work, you can find Greg spending time with his family, skiing, hiking, biking, playing on the water, and voraciously consuming words (reading).

eBPF (extended Berkeley Packet Filters) is a modern kernel technology that can be used to introduce dynamic tracing into a system that wasn't prepared or instrumented in any way. The tracing programs run in the kernel, are guaranteed to never crash or hang your system, and can probe every module and function -- from the kernel to user-space frameworks such as Node and Ruby.

In this workshop, you will experiment with Linux dynamic tracing first-hand. First, you will explore BCC, the BPF Compiler Collection, which is a set of tools and libraries for dynamic tracing. Many of your tracing needs will be answered by BCC, and you will experiment with memory leak analysis, generic function tracing, kernel tracepoints, static tracepoints in user-space programs, and the "baked" tools for file I/O, network, and CPU analysis. You'll be able to choose between working on a set of hands-on labs prepared by the instructors, or trying the tools out on your own test system.

Next, you will hack on some of the bleeding edge tools in the BCC toolkit, and build a couple of simple tools of your own. You'll be able to pick from a curated list of GitHub issues for the BCC project, a set of hands-on labs with known "school solutions", and an open-ended list of problems that need tools for effective analysis. At the end of this workshop, you will be equipped with a toolbox for diagnosing issues in the field, as well as a framework for building your own tools when the generic ones do not suffice.

Sasha Goldshtein is the CTO of SELA Group, a Microsoft C# MVP, and a Pluralsight author. He leads the Performance and Debugging team at SELA Technology Center, and is the author of numerous training courses, open source projects, books, and online articles on diagnostic tools and performance optimization. Sasha consults on various topics, including production debugging, application and system troubleshooting, performance investigation, and distributed architecture.

From stores with ads in the Super Bowl to selling Kanye’s latest album, Shopify has built a name for itself handling some of the world’s largest flash sales. These high profile events generate write-heavy traffic that can be four times our platform’s baseline throughput and don’t lend themselves to off-the-shelf solutions.

This talk is the story of how we engineered our platform to survive large bursts of traffic. Since it’s not financially sound for Shopify to have the required capacity always running, we built queueing and page caching layers into our Nginx load balancers with Lua. To guarantee these solutions worked, we tested them with a purpose-built load testing service.

Although flash sales are unique to commerce platforms, the lessons we learn from them are applicable to any services that experience bursts of traffic.

Emil is a production engineer at Shopify where he works on performance, the production pipeline, and DNS tooling. When he's not trying to make Shopify's global performance heat map green, he's shivering over a spiked cup of coffee in the great Canadian north.

If Rip Van Winkle had gone to sleep around 2006 and woken up 10 years later, he'd find the world a strange brew of the new and the old. He'd be amazed that phones had grown a brain, dismayed that a most excellent rendition of the Dark Knight had wandered back to the wasteland as most Dark Knight capers do. People had warmed upto electric cars, but not to climate change. And, if Ol' Rip were a network operations guy at some of the large webscale companies, he might think he'd died and woken up in heaven. Networks were no longer slow as molasses: to deploy, manage and upgrade. He'd find some things had stayed the same (IPv4 still ruled the roost), and some others not so much. He would be puzzled by the terminology and the discussions as he wandered the hallways. SDN, Open networking, Openflow, microservices, Ansible, Puppet, Kubernetes, and so on.

This tutorial is an attempt to bring folks up to speed on whats happened with networking in the past 10 years or so, especially in the data center, concluding with some thoughts on why exciting times lie ahead. The talk will be roughly divided into the following sections:

1. Who Moved My Network ? What's causing all this turmoil in networking
2. Solutions: Requirements, Terminology, Pros and Cons
3. Changing Landscape: Network Topologies
4. Changing Foundation: Network Protocols
5. Changing Operations: Modern Operations
6. Changing Residents: Modern applications and their implications on networks
7. Reading Tea Leaves

The tutorial will include demos and hands on work with some modern tools.

The audience is expected to be aware of basic networking (bridging, routing, broadcast, multicast etc.).

The key takeways from this talk will be:

• An understanding of the forces behind the changes in data center networking
• The morphology an physiology of modern DC networks
• What these changes presage of the future

Some preliminary ideas for hands on work:

• Build multi-host container network
• Build and configure a nxm CLOS topology with BGP
• Design a CLOS for x number of servers given certain box specifications

Dinesh G. Dutt has been in the networking industry for the past 15 years, most of it at Cisco Systems. Before joining Cumulus, he was a Fellow at Cisco Systems. He has been involved in enterprise and data center networking technologies, including the design of many of the ASICs that powered Cisco's mega-switches such as Cat6K and the Nexus family of switches. He also has experience in storage networking from his days at Andiamo Systems and in the design of FCoE. He is a co-author of TRILL and VxLAN, and has filed for over 40 patents.

Have you ever bought machines? What if you need to even build datacenters? How can you predict how many you are going to need in two years from now? How can you make efficient use of all the resources you suddenly got? What if you are missing some resources? Can we automate all these stuff and integrate with our continuous delivery?

These are just a few questions anyone planning a large computer fleet always make. This talk will cover some of the approaches and tooling that can be used to effectively plan for the demand of services and how to cover it on the most efficient manner.

Ramón has been an SRE Technical Lead on Identity and Authentication services at Google since 2013.

If Rip Van Winkle had gone to sleep around 2006 and woken up 10 years later, he'd find the world a strange brew of the new and the old. He'd be amazed that phones had grown a brain, dismayed that a most excellent rendition of the Dark Knight had wandered back to the wasteland as most Dark Knight capers do. People had warmed upto electric cars, but not to climate change. And, if Ol' Rip were a network operations guy at some of the large webscale companies, he might think he'd died and woken up in heaven. Networks were no longer slow as molasses: to deploy, manage and upgrade. He'd find some things had stayed the same (IPv4 still ruled the roost), and some others not so much. He would be puzzled by the terminology and the discussions as he wandered the hallways. SDN, Open networking, Openflow, microservices, Ansible, Puppet, Kubernetes, and so on.

This tutorial is an attempt to bring folks up to speed on whats happened with networking in the past 10 years or so, especially in the data center, concluding with some thoughts on why exciting times lie ahead. The talk will be roughly divided into the following sections:

1. Who Moved My Network ? What's causing all this turmoil in networking
2. Solutions: Requirements, Terminology, Pros and Cons
3. Changing Landscape: Network Topologies
4. Changing Foundation: Network Protocols
5. Changing Operations: Modern Operations
6. Changing Residents: Modern applications and their implications on networks
7. Reading Tea Leaves

The tutorial will include demos and hands on work with some modern tools.

The audience is expected to be aware of basic networking (bridging, routing, broadcast, multicast etc.).

The key takeways from this talk will be:

• An understanding of the forces behind the changes in data center networking
• The morphology an physiology of modern DC networks
• What these changes presage of the future

Some preliminary ideas for hands on work:

• Build multi-host container network
• Build and configure a nxm CLOS topology with BGP
• Design a CLOS for x number of servers given certain box specifications

Dinesh G. Dutt has been in the networking industry for the past 15 years, most of it at Cisco Systems. Before joining Cumulus, he was a Fellow at Cisco Systems. He has been involved in enterprise and data center networking technologies, including the design of many of the ASICs that powered Cisco's mega-switches such as Cat6K and the Nexus family of switches. He also has experience in storage networking from his days at Andiamo Systems and in the design of FCoE. He is a co-author of TRILL and VxLAN, and has filed for over 40 patents.

Gathering telemetry data is key to operating reliable distributed systems at scale. Once you have set up your monitoring systems and recorded all relevant data, the challenge becomes to make sense of it and extract valuable information, like:

• Is the system down?
• Is user experience degraded for some percentage of our customers?
• How did our query response times change with the last update?

Statistics is the art of extracting information from data. In this tutorial, we address the basic statistical knowledge that helps you at your daily work as an SRE. We will cover probabilistic models, summarizing distributions with mean values, quantiles, and histograms and their relations.

The tutorial focuses on practical aspects, and will give you hands-on knowledge of how to handle, import, analyze, and visualize telemetry data with UNIX tools and the IPython toolkit.
This tutorial has been given at several occasions over the last year and has been refined and extended since, cf. Twitter #StatsForEngineers

Imagine you're tackling one of these evasive performance issues in the field, and your go-to monitoring checklist doesn't seem to cut it. There are plenty of suspects, but they are moving around rapidly and you need more logs, more data, more in-depth information to make a diagnosis. Maybe you've heard about DTrace, or even used it, and are yearning for a similar toolkit, which can plug dynamic tracing into a system that wasn't prepared or instrumented in any way.

Hopefully, you won't have to yearn for a lot longer. eBPF (extended Berkeley Packet Filters) is a kernel technology that enables a plethora of diagnostic scenarios by introducing dynamic, safe, low-overhead, efficient programs that run in the context of your live kernel. Sure, BPF programs can attach to sockets; but more interestingly, they can attach to kprobes and uprobes, static kernel tracepoints, and even user-mode static probes. And modern BPF programs have access to a wide set of instructions and data structures, which means you can collect valuable information and analyze it on-the-fly, without spilling it to huge files and reading them from user space.

In this talk, we will introduce BCC, the BPF Compiler Collection, which is an open set of tools and libraries for dynamic tracing on Linux. Some tools are easy and ready to use, such as execsnoop, fileslower, and memleak. Other tools such as trace and argdist require more sophistication and can be used as a Swiss Army knife for a variety of scenarios. We will spend most of the time demonstrating the power of modern dynamic tracing -- from memory leaks to static probes in Ruby, Node, and Java programs, from slow file I/O to monitoring network traffic. Finally, we will discuss building our own tools using the Python and Lua bindings to BCC, and its LLVM backend.

Sasha Goldshtein is the CTO of SELA Group, a Microsoft C# MVP, and a Pluralsight author. He leads the Performance and Debugging team at SELA Technology Center, and is the author of numerous training courses, open source projects, books, and online articles on diagnostic tools and performance optimization. Sasha consults on various topics, including production debugging, application and system troubleshooting, performance investigation, and distributed architecture.

Gathering telemetry data is key to operating reliable distributed systems at scale. Once you have set up your monitoring systems and recorded all relevant data, the challenge becomes to make sense of it and extract valuable information, like:

• Is the system down?
• Is user experience degraded for some percentage of our customers?
• How did our query response times change with the last update?

Statistics is the art of extracting information from data. In this tutorial, we address the basic statistical knowledge that helps you at your daily work as an SRE. We will cover probabilistic models, summarizing distributions with mean values, quantiles, and histograms and their relations.

The tutorial focuses on practical aspects, and will give you hands-on knowledge of how to handle, import, analyze, and visualize telemetry data with UNIX tools and the IPython toolkit.
This tutorial has been given at several occasions over the last year and has been refined and extended since, cf. Twitter #StatsForEngineers

Motivated by the problem of predicting whether any given ad would be clicked in response to a query, in this introductory talk we outline the requirements and large-system design challenges that arise when designing a machine learning system that makes millions of predictions per second with low latency, learns quickly from the responses to those predictions, and maintains a consistent level of model quality over time. We present alternatives for meeting those challenges using diagrams of machine learning pipelines.

Concepts used in this talk: machine learning (classification), software pipelines, sharding and replication, map-reduce

Graham is an SRE at Google working on machine learning pipelines for ad click prediction.  

This tutorial/workshop is aimed at management and individual contributors alike. We will work together on how to encourage and nurture a diversity culture in day-to-day ops teams. First we will discuss the concepts of 2- and 3-dimensional diversity, and the statistics around diverse teams performance. Then we will map out how to design, build, deploy and operate a diversity plan in our teams. This will include diversity goal setting and explicit cultural evolution, hiring processes, day to day communications, review process and team collaboration. Where possible we will encourage groups to break out and evaluate their own cultures and processes.

This tutorial/workshop is aimed at management and individual contributors alike. We will work together on how to encourage and nurture a diversity culture in day-to-day ops teams. First we will discuss the concepts of 2- and 3-dimensional diversity, and the statistics around diverse teams performance. Then we will map out how to design, build, deploy and operate a diversity plan in our teams. This will include diversity goal setting and explicit cultural evolution, hiring processes, day to day communications, review process and team collaboration. Where possible we will encourage groups to break out and evaluate their own cultures and processes.

It's easy to plan out month-long or year-long projects, or to have an interrupts rotation for dealing with oncall/tickets, but how do you make sure you're also doing the short week-long projects that can relieve your technical debt? I'll cover a planning approach that my team found that makes room for all three sets of work, reducing in the long term the operational burden of the services we operate.

Liz is a Senior Site Reliability Engineer at Google and manages a team of SREs responsible for Google's storage systems. She lives with her wife, metamour, and two Samoyeds in Brooklyn. In her spare time, she plays classical piano, leads an EVE Online alliance, and advocates for transgender rights.

Docker is very popular these days, but how many of us are really familiar with the basic building blocks of Linux containers and their implications? What's missing in the good ol’ chroot jails? What are the available Copy-on-Write options and what are their pros and cons? Which syscalls allow us to manipulate Linux namespaces and what are their limitations? How do resource limits actually work? What different behaviours do containers and VMs have?

In this hands-on workshop, we will build a small Docker-like tool from O/S level primitives in order to learn how Docker and containers actually work. Starting from a regular process, we will gradually isolate and constrain it until we have a (nearly) full container solution, pausing after each step to learn how our new constraints behave.

Avishai Ish-Shalom is a veteran Ops and a survivor of many production skirmishes. Avishai helps companies deal with web era operations and scale as an independent consultant. In his spare time Avishai is spreading weird ideas and conspiracy theories such as DevOps.

Fuzz testing has been used to evaluate the robustness of operating system distributions for over twenty years. Eventually, a fuzz test suite will suffer from reduced effectiveness. The first obstacle is the pesticide paradox: as you fix the easy defects, it gets difficult to find the remaining obscure defects. Also, the test execution time and the debug/fix cycle tends to be manual work that can take hours or even days of effort. During the presentation, a structured framework for creating new fuzz tests will be introduced, along with a competitive analysis approach used to minimize defect reproduction complexity.

Kirk is currently a Production Engineer at Shopify, making sure that our docker image build system can keep up with 22 launches a day.

Docker is very popular these days, but how many of us are really familiar with the basic building blocks of Linux containers and their implications? What's missing in the good ol’ chroot jails? What are the available Copy-on-Write options and what are their pros and cons? Which syscalls allow us to manipulate Linux namespaces and what are their limitations? How do resource limits actually work? What different behaviours do containers and VMs have?

In this hands-on workshop, we will build a small Docker-like tool from O/S level primitives in order to learn how Docker and containers actually work. Starting from a regular process, we will gradually isolate and constrain it until we have a (nearly) full container solution, pausing after each step to learn how our new constraints behave.

Avishai Ish-Shalom is a veteran Ops and a survivor of many production skirmishes. Avishai helps companies deal with web era operations and scale as an independent consultant. In his spare time Avishai is spreading weird ideas and conspiracy theories such as DevOps.

For several years, Google's internal surveys identified the lack of trustworthy, discoverable documentation as the #1 problem impacting internal developer productivity. We're not alone: Stack Overflow's 2016 survey ranked ""Poor documentation"" as the #2 problem facing engineers.

Solving this problem is tough. It's not enough to build tooling; the culture needs to change. Google internal engineering is attacking the challenge three ways: Building a documentation platform; integrating that platform into the engineering toolchain; and building a culture where documentation - like testing - is accepted as a natural, required part of the development process.

In this talk, we'll share our learnings and best practices around both tooling and culture, the evolution of documentation, and some thoughts about how we can transition from the creation of documents towards an ecosystem where context-appropriate, trustworthy documentation is reliably and effortlessly available to the engineers that need it.

Riona MacNamara is a staff technical writer at Google. She leads Google's Documentation Infrastructure team, which aims to make internal engineers happier and more productive by fully integrating the creation, maintenance, and discovery of engineering information into our development workflow and culture. She previously worked at Amazon and Microsoft.

Docker is very popular these days, but how many of us are really familiar with the basic building blocks of Linux containers and their implications? What's missing in the good ol’ chroot jails? What are the available Copy-on-Write options and what are their pros and cons? Which syscalls allow us to manipulate Linux namespaces and what are their limitations? How do resource limits actually work? What different behaviours do containers and VMs have?

In this hands-on workshop, we will build a small Docker-like tool from O/S level primitives in order to learn how Docker and containers actually work. Starting from a regular process, we will gradually isolate and constrain it until we have a (nearly) full container solution, pausing after each step to learn how our new constraints behave.

Avishai Ish-Shalom is a veteran Ops and a survivor of many production skirmishes. Avishai helps companies deal with web era operations and scale as an independent consultant. In his spare time Avishai is spreading weird ideas and conspiracy theories such as DevOps.

In 2013, Mikey Dickerson joined what became known as the “ad hoc” team, tasked with rescuing HealthCare.gov after its failed launch on October 1. In August 2014, President Obama established the United States Digital Service and appointed Mikey to serve as the Administrator to see if the strategy that succeeded at pulling Healthcare.gov out of the fire could be applied to other government problems. Now nearly 2 years old and about 150 people spanning a network of federal agencies, the U.S. Digital Service has taken on immigration, education, veterans benefits, and health data interoperability. The U.S. Digital Service is helping agencies build effective government services and improve IT procurements by focusing on industry best practices and agile methodology, ultimately driving change in the largest institution in history. Prior to joining the U.S. Digital Service, Mikey worked as a Site Reliability Manager at Google.