SREcon22 Europe/Middle East/Africa Conference Program

This talk shares the findings from a series of exploratory discussions between two prominent Site Reliability Engineers from industry-leading organizations and two socio-technical systems researchers with extensive experience in distributed human-machine teaming.

Drawing from both academic and industry perspectives, this talk elaborates on topics relevant to socio-technical software systems - both practical considerations and philosophical concerns. The practical includes: the tradeoffs inherent in balancing operational load and work in support of feature delivery with the less-immediately-tangible - but no less important - work of learning about our systems; sharing knowledge as a team and; using that knowledge to reduce risk. Our philosophical inquiries relate to the impact of the history of SRE on its future, meditations on the ‘practice’ and values of SRE and provocative promising new directions.

Attendees will come away with new perspectives on how knowledge and power structures operate in their organizations, shaping the ways that we conduct and understand our work.

It’s no secret that in the SRE community, for better or worse, we often look up to large companies (almost always Google), on how we should go about building secure, reliable & maintainable systems. Lots has been said about what they do right but in reality, these approaches are rarely applicable to SREs at smaller organizations who just don’t have the resources to replicate the same systems.

Here’s the thing - it’s not just the core technology that ex-Google SREs miss the most: it’s all the little things. How do you make on-boarding not suck? What’s an effective way of securing systems, but not getting in the way of people doing their jobs? And who the hell ever thought JIRA was a good idea?

This talk is structured as a comedy double act: a recovering Google Engineer seeks therapy from someone who’s lived their whole career outside the walled garden that is Google.

As a profession, SRE is still in its infancy, but it isn't too young to be experiencing a profound identity crisis.

Google's Ben Treynor originally said that SRE is what you get when you ask software engineers to do operations (but recently contradicted himself by saying that SRE isn't operations). Google later described SRE as an implementation of DevOps. Lots of people see SRE as holding the pager, or as use of SLOs.

SRE is bigger than any of the definitions above. I define SRE as systems thinking applied to software in production, including the sociotechnical aspects of running software systems. Systems thinking has a set of associated tools and methodologies, but, at its core, it is a philosophy that is aware that systems have underlying structures that cause particular patterns of behaviour. Systems thinking aims to model complex problems and make them tractable.

The best SREs and the best SRE organisations, are thoroughly immersed in systems thinking but it largely remains an implicit organising principle, not something we make a first-class citizen in our profession.

Niall Murphy, at SREcon 2021, asked what SRE 2.0 should be. My answer is that SRE 2.0 should be a profession where we put systems thinking first. This talk will explore how we might do that.

Migrations are one of the most challenging tasks we do as infrastructure engineers. At Slack, we switched from HAProxy to Envoy Proxy. Overall, this migration was a success, and did not cause any downtime, but even so, we ran into several interesting edge cases that caused minor problems.

Troubleshooting these sorts of 'gray' failures can be a difficult technical challenge. So this talk will discuss some of those facepalm moments: how they were detected, steps taken to investigate them, and how they were solved.

Takeaways from this talk include a specific set of approaches for debugging such problems with Envoy Proxy and other web proxies that we learnt via these events along with some engineering practices that eases the stress during a large migration.

Most SREs are "product" SRE; concerned with customers, external to your organisation. They care about latency, reliability, ensuring you have capacity enough to keep your customers happy. Most product teams work in human-reaction-speed latency; 30 to 500 milliseconds. They direction from a VP somewhere that moves a lever between RELIABILITY and VELOCITY while trying to work out how to make both better. Happy product SREs are able to ship changes multiple times an hour.

A good number of SRECon folk are "infrastructure" SRE. The doozers who ensure that their product teams spend most of their time caring about external matters, not internals that don't add customer value. This talk explores why that world is different, and why those SREs need different skills.