SREcon17 Europe/Middle East/Africa Conference Program

The SRE discipline necessitates deep understanding of the organisation's business and technical needs, as well as clear objectives across the entire team.

What happens when you begin to distribute your SRE team across a large geological divide?

We'll talk about what went into creating Wayfair's global SRE presence, from initial hire to full contributing team—what worked, what didn't, and what we have planned.

Telemetry monitors and their (constant) beeping is a pretty common sight in hospitals. I saw these at the NICU where my twins were being cared for after being born prematurely. My wife and I used to freak out every time one of these went off. Unlike a missed alarm that said your site's down, failing to act on an alarm at a hospital can have much more critical consequences; in 2010 at a hospital in Massachusetts, a patient's death was directly linked to telemetry monitoring after alarms signaling a critical event went unnoticed by 10 nurses.

I attempted to solve this problem when I joined Zynga (in 2013) as the head of SRE. I will go over our failed attempts including filtering the noise, adding heads, building more tools, etc. Will also cover how I came up with an initiative called "clean room" as a way to incentivize engineering teams to keep the noise levels low. Finally, go over some of the tactics that worked (and ones that didn't).

Most people I spoke to about "clean room" almost always walked away having learned something (some have said it's common sense). Share, learn, ask questions, participate - I'll try to make it fun!

This talk gives an inventory and overview of the different methods for dealing with load-shedding and overload in production stacks, including an overview of the methods developed at Google and the open-source solutions.

We'll review the pros and cons, scope and effort levels of each method, and compare with existing approaches, including circuit-breakers.

Shipping code is not enough. You also need logs, tests and static analyzers to have the necessary confidence in the change you just deployed. Especially when you’re shipping to production every 10 minutes, at peak time.

A common philosophy for feeling that things are in control is simply adding more data. And when something bad happens, first reaction is even more instrumentation to cover that specific scenario. You need more information in the logs to troubleshoot, more inputs for your tests and more linting rules. And then you’ll never run into that problem again, right!?

Well…maybe, but you’ve just hit a bigger one:

When your app is small, you can easily get away with test duplication, log noisiness and introducing new tools.

As you grow, the noise becomes impactful. You’re being slowed down by the shortcuts and poor decisions you took earlier, until it becomes a non trivial problem to solve.

In this talk we will explore the approach we took in Intercom for introducing information sanity, by focusing on:

• logging myths, and dangerous fallacies
• being deliberate about operational and engineering needs
• better troubleshooting using structured and canonical logs
• getting more benefits and performance out of notoriously slow static analyzers
• writing tests with performance in mind

All of us depend on the underlying network to be stable whether in the datacenter or in the cloud. We all have a basic knowledge of how traditional networks run, however in the past 10 years, we’ve moved to building redundant physical topologies in our networks, optimized the routing methodologies accordingly, moved into the cloud and gotten greater visibility and tuneables in the Linux kernel network stack. A lot has changed!

However, the way we troubleshoot the network in relation to the applications we support hasn’t adapted. In this session, we’ll review the progress that network infrastructure has made look at specific examples where traditional troubleshooting responses fail us and demonstrate our need to rethink our approach to making applications and the network interact harmoniously.

Large organisations often have teams dedicated to building and using load test frameworks for their production services. Intercom's engineering team was too small to have accurate load tests for all of it's systems, but as we acquired larger customers, having accurate load-test numbers, and being able to communicate them to the business became more critical. This talk will cover some things we learnt about load testing, and how it changed our mental models of some of our infrastructure.

In a world with ever-growing DDoS attacks, L7 attacks give even the most experienced engineers the sweats. Imagine if instead of following easy to detect patterns, bots could mimic the behaviour of customers. Well, that’s exactly what Shopify sees every day during flash sales.

Come and learn how we block nearly all bot traffic on our load balancers without any human intervention. We will share our challenges of differentiating between web crawlers and bots, users behind NATs and bots rotating user agents, as well as fast humans and browser extensions. When the stakes are blocking a customer completing a checkout, misclassification isn’t an option.

This is not yet another machine learning talk, but an example of how simple statistics, heuristics and some sane limits can give great results with minimal complexity. The lessons learned in this talk are applicable to any real-world problem with inexact constraints.

By now, most of us have read the O’Reilly book about Google SRE or have heard of other tech companies’ SRE teams. Our story is about doing SRE in a more traditional and more regulated environment: in the largest bank in the Netherlands.

This talk will address the history, present and future of SRE within a financial institution with a BizDevOps way of working. In doing so, we will talk about how our journey started at SREcon in Dublin last year, why we wanted and need to do SRE within ING, how our SREs started, the distributed way of working of SRE within ING, what technologies we actually work with and why, and our plans for the future.

Lastly, we will share our SRE dos-and-donts after a year of experience. We hope our talk inspires engineers or tech leads to start doing SRE within their (possibly more traditional) company and that those who already implemented SRE can learn from our journey.

You decoupled your APIs from their implementations and put them behind RPC interfaces. You build and deploy services independently. You code health is impeccable. You put your user data in a persistent, replicated, and consistent store, where it belongs. Your developer velocity has skyrocketed.

Now we have new problems. We’ve got N independent services with M edges of interaction between them. That’s N services that need to be built, tested, and deployed on the infrastructure that expected you to have one service whose mess of entanglement was a secret you had with the compiler.

How do we deploy N binaries with N sources of static configuration and M sources of runtime configuration safely without losing our collective minds? In this talk, I’ll share some of how we grew that aforementioned N from 1 to many in Gmail. Specifically:

• Consistent naming schemas for services, environments -
• Maintaining lightweight, easy-to-change production configuration abstraction layers
• Release early, often
• Canary everything by sharding into more A/B environments than you'd think you’d need 
• Encourage backwards compatibility in all APIs 
• Validate and test all configuration before changing global state

And, of course, some of things we (Gmail) learned by breaking things along the way.

Getting customers to care about Reliability is hard. Getting stakeholders to care about Reliability is harder. Getting the entire company to care about Reliability is even harder.

In this talk, I will cover what steps that every leader in any organization can take to get more people to care about Reliability. Because Reliability is one of those things that people only notice when it goes in the wrong direction, it can be hard to show the value of it and why it is so important.

We will walk through cultural and management changes, metrics to watch and obsess over, and some tooling that can help along the way.

Rollout automation is something that every service and team needs, and many reinvent the wheel. I'll talk about - why the wheel gets reinvented - a system design that discourages reinvention, including an architecture diagram - the organisational challenges encountered when converting many services to use this new system design - how well the conversion attempt worked in practice This is based on my experience initiating and running a program to replace rollout automation across Storage SRE in Google.

Agilent Technologies, while traditionally known as a hardware company, has started to deliver several Software-as-a Service offerings through acquisitions and in-house product development. This talk will discuss how to transform the thinking across a decentralized, large corporate organization to approach software delivery differently. Traditionally, we are used to delivering software by burning and shipping CDs/DVDs with annual release cycles. Fast-forward to today with SaaS products using CI/CD approaches, etc and releasing as frequently as twice a week, this requires a completely different operating model and doesn’t fit in the traditional organizational framework/processes.

What happens when you take the leap of faith and leave the security of a systems engineering team to become the first systems person at a startup? What should you expect?

This talk is about the challenges of being the sole systems engineer at a young company. The amount of work is overwhelming but the experience is worth it. We will explain the key elements of a newborn infrastructure and the stages leading it to maturity.

The challenges of this role are not limited to solving technical problems. The habits, processes and standards you will establish, will pave the way to go from a single engineer to a team.

This presentation replaces the talk by Dinah McNutt, who is unable to attend. Laura Nolan will revisit her SRE 101 content from yesterday; if you missed the session due to the meeting room being at capacity, this is your opportunity to attend.

The purpose of an SRE team is to keep its services up, reliable, performant and efficient. How do effective SRE teams do this?

We'll run through an overview of key SRE competencies: monitoring and alerting, incident response, disaster recovery, performance and efficiency, change management and capacity planning.

We'll also look at the habits of successful SRE teams and some common pitfalls.

It's very easy to launch a debugger on your dev box, attach to the right process and step through code. However, things are different when you need to debug an issue in production that's getting tens of thousands of requests per second. What if the issue reproduces only in production? How do you debug without affecting production traffic? What techniques can you use in your development to make it easier to debug issues? Does your application use tracing? What debug logs are written out to aid in analysis?

This talk will cover:

1. Challenges with debugging in production
2. Various approaches that are used in the industry
3. Examples from Bing & Cortana incidents and steady state problems to illustrate the techniques
4. Service design ideas that make them easier to debug

The talk will feature research from international standardisation committees, the history of women, work, and computerization, and personal anecdotes of the mainframe era to construct a theory as to how and why engineering and operations split apart. We will also examine the issues with an eye to gender and workplace politics.

In the course of this talk, we will look at:

• The earliest jobs in computers (~40s). What were they, why did they exist, and who held them?
• What different and distinct things happened in the 50s, 60s, 70s, 80s, 90s, and noughties?
• What was the difference between the keypunch operator and the computer operator?
• What was unglamorous about programming in the 40s, such that women were allowed to do it?
• What factors led to women being crowded out of programming as a profession?
• Why did system administration come into being as a job family, and why was it minimally viable to separate that from programming?

All of us have seen these evasive performance issues or production bugs in the field, which standard monitoring tools don't see or catch. BPF is a Linux kernel technology that enables fast, safe, dynamic tracing of a running system without any preparation or instrumentation in advance. The JVM itself has a myriad of insertion points for tracing garbage collections, object allocations, JNI calls, and even method calls with extended probes. When the JVM tracepoints don't cut it, the Linux kernel and libraries allow tracing system calls, network packets, scheduler events, off-CPU time, time blocked on disk accesses, and even database queries. In this talk, we will see a holistic set of BPF-based tools for monitoring JVM applications on Linux, and revisit a systems performance checklist that includes classics like fileslower, opensnoop, and strace—all based on the non-invasive, fast, and safe BPF technology.

The sparsely-attended SREcon17 Americas Tech Writing talk focused on HOW to work with Tech Writers. I'm instead focusing on WHY you should work with a TW--because they make your life easier, and can make the work you're already doing have more impact.

Reasons to engage a TW:

• If you need to explain your product/service/etc. to users: Chances are, the most satisfying, enjoyable, and rewarding part of your job is engineering work—creating a tool, fixing a problem, redesigning infrastructure, etc. It *isn't* answering the same questions from users over and over. → Get solid documentation in place to free up engineer time.
• If your team internal documentation is a mess: It might be hard to find the docs you need when you get paged, hard to identify current content, or some information might just be flat-out missing. → A TW can help you whip a documentation rat's nest into shape, and give you the tools to maintain your docs easily moving forward.
• If you want to make your work more visible (so other people can leverage it and learn from it): → A TW can help you get that information out there!

What data do you hold?
Are you processing the data, or controlling it?
Do you have the consents to use that data like that?
Do you have a register of all that data and every way you use it, and what for?
Can you find every piece of data you hold that relates to an individual, copy it and send it to them—for free—within 30 days?
What happens when they say they want it erased?

The General Data Protection Directive comes into force on the 25th May 2018. New powers mean regulators can impose fines for breaches up to 4% of annual turnover. This workshop is for anyone trying to make sure that their organisation isn't in breach by the implementation date.

GDPR isn't just a compliance project. It's a business culture change project. Let's struggle our way through together.

Awareness and discussion of psychological safety as a key ingredient for productive and successful teams has grown recently, thanks to media coverage and pioneering research by companies and scholars. While flagrant forms of disrespect like angry shouting and insults obviously threaten psychological safety in teams, so too can passive-aggressive behaviors such as complaining, pouty silence, “forgetting” to complete tasks, and stubbornness.

In the SRE context, passive-aggressive behaviors can have disastrous consequences. These include outages and incidents that could have been avoided with better preparation or notification; narrowly focused quick-fixes instead of systemic, long-term maintenance efforts; blame instead of solutions-oriented post-mortems, and refusal to share knowledge. In many cases, few or no words are spoken; silent resistance is the hostile act.

This talk will bring attention to passive-aggressive behavior as a set of hostile acts that one can (and should) identify, manage and overcome in the tech/SRE environment. For context, it will draw upon psychological research, history, a bit of pop culture for fun, and anecdotes from SREs. And for guidance, you’ll hear some tried-and-true agile and communications methods for managing or eliminating passive-aggressive behavior in your teams and interactions.

What data do you hold?
Are you processing the data, or controlling it?
Do you have the consents to use that data like that?
Do you have a register of all that data and every way you use it, and what for?
Can you find every piece of data you hold that relates to an individual, copy it and send it to them—for free—within 30 days?
What happens when they say they want it erased?

The General Data Protection Directive comes into force on the 25th May 2018. New powers mean regulators can impose fines for breaches up to 4% of annual turnover. This workshop is for anyone trying to make sure that their organisation isn't in breach by the implementation date.

GDPR isn't just a compliance project. It's a business culture change project. Let's struggle our way through together.

In this presentation we'll re-examine monitoring to understand how to formulate valuable goals and align monitoring design and implementation with those goals. With a focus on outcomes and behavior that leads to outcomes we'll focus on performance data and not security monitoring.

Attendees will learn to ask the right questions when approaching the monitoring of systems and businesses. They will understand why and how monitoring should fit into the overall systems architecture to reduce risk and increase value.

The Cloud is a scary place. You're trusting your entire business to a platform out of your control. This applies to any platform, any SaaS, PaaS, IaaS provider. If they have an outage it's out of your hands!

Introducing CRE: Expanding SRE to inside your customer's organisation. Making a reliable system for your own business is one thing, but doing it when you are providing a platform to another company is a new and exciting area.

Learn how Google is exploring Customer Reliability Engineering. Sharing everything from design decisions to monitoring.