SREcon23 Asia/Pacific Conference Program

Batteries play a critical role in providing uninterrupted power during outages, but managing their lifecycle can be challenging and has environmental impact. In this presentation, we will introduce a sustainable solution for managing the lifecycle of batteries in data center infrastructure. Our solution employs IoT sensors, mechanism models, and AI models to monitor battery performance in real-time, analyze health, detect anomalies, and predict end-of-life. Real-time monitoring dashboards visualize battery performance and behaviors, while health analysis capabilities detect anomalies and predict failures. The solution recommends proactive maintenance to prevent costly downtime and triggers sustainable waste management processes. It reduces costs and environmental impact towards zero-carbon waste diversion goals. We will showcase real-world examples of our solution and how to improve waste diversion. The solution is deployed and running in production, supporting daily operations.

In a brownfield, the transition to immutable infrastructure can take years of concentrated effort; leaving teams with a complex set of mutable infrastructure and technical debt that requires a robust set of tools and processes to support as the architecture evolves. Managing these tools and processes across a massive hybrid cloud spanning more than 5,000 compute instances across 12+ data centers and public cloud is a daunting prospect for any SRE team.

I’ll share my story of evolving deployment infrastructure in this world from a developer-centric and push-oriented model to a distributed, eventually consistent, and pull-oriented model despite the challenges associated with traditional infrastructure and technical debt. I’ll also share what I’ll be keeping an eye out for in the future and how teams with similar problems but different environments can apply my design concepts to solve the problems and technical debt they face in their own work.

The Linux Kernel lies at the heart of many high profile services and applications. And since the kernel code executes at the highest privilege level it is very important to keep up with kernel updates to ensure the production systems are patched in a timely manner for numerous security vulnerabilities discovered almost every day.

Yet, because the kernel code executes at the highest privilege level and a kernel bug usually crashes the whole system, many SREs, production engineers and system administrators try to avoid upgrading the kernel too often just for the sake of stability. In many companies we have seen a tendency to create more obstacles to Linux kernel releases (requiring more approvals, harder update justifications, requiring more time in canary testing etc). But introducing all these obstacles and not treating kernel updates like any other software updates usually significantly increases the risk for the company and their service of being exploited.

One of the reasons SREs and production engineers are too afraid of ANY kernel upgrade is that they don’t actually know the details about Linux kernel release process and policy. This talk tries to demystify Linux Kernel releases and provides a guide on how to distinguish a kernel bugfix release from a feature release. We also try to explore why commonly established perceptions and patterns around production kernel releases are wrong and how you actually risk the stability of your systems by not releasing the kernel regularly. In the end we describe how kernel releases are implemented in our company and propose possible approaches to deploy kernel upgrades regularly with minimal risk.

There's one skill that has been a large part of the success I've had as an SRE and it has nothing to do with technical knowledge or understanding the inner workings of Kubernetes. It’s my soft skills and how I work with others. But what even are "soft skills"? How do you develop those skills if you don’t have them? How do you get better at "working with people"?

This talk will use personal and professional anecdotes to explain the concepts behind "soft skills" and give you real usable advice and drills to develop these skills - regardless of whether you are an introvert or an extrovert. I’ll break it down and deliver it in context for technical professionals and walk through why I think these skills and concepts are such an integral part of the SRE discipline.

Platform and Site Reliability Engineering is just as crucial for startups as it is for bigger organizations, if not more, as it creates enabling technologies and culture for the various Engineering, Product Development, and Business use cases of organizations, and imbibes a culture of continuous feedback. SRE, done "suitably" right, can lead companies to efficiently deliver high-quality, secure, compliant, robust, and reliable products to their customers. This talk proposes to share the lessons learned from setting up teams and building Platforms and SRE culture for Startups and Scaleups. After the talk, the audience will take away a Framework of principles for setting up a Culture of Site Reliability Engineering, and practical learnings and insights to build and scale SRE teams and platforms that can handle the demands of a growing user base, in a dynamic and fast-changing startup environment.

The industry defined as good practice to have as few alerts as possible, by alerting on symptoms that are associated with end-user pain rather than trying to catch every possible way that pain could be caused.

Organizations with complex distributed systems that span dozens of teams can have a hard time following such practice without burning out the teams owning the client-facing services. A typical solution is to have alerts on all the layers of their distributed systems. This approach almost always leads to an excessive number of alerts and results in alert fatigue.

Adaptive Paging is an alert handler that leverages the causality from tracing and Opentracing/OpenTelemetry's semantic conventions to page the team closest to the problem. From a single alerting rule, a set of heuristics can be applied to identify the most probable cause, paging the respective team instead of the alert owner.

The approach enables an effective symptom-based alerting strategy with thresholds derived from the respective operation service level objective.

In 2020, inspired after attending the Oct 2019 SRECon in Dublin, we embarked on our own SRE (and data engineering) transformation. Now 3 years later we would like to share our story transforming 500+ Production facing engineers from a process driven way of working to a dynamic, award winning, software engineering team who place SRE at their heart. The session will include a detailed review of the "how did we do it", lessons learned, key enablers that accelerated our transformation, and ongoing challenges – hopefully with some crowd sourced solutions! Key topics will be People and Culture, Observability, and measuring client experience with a 360 degree lens using quantitative and qualitative indicators.