SREcon22 Asia/Pacific Conference Program

Despite having world class reliability systems, some incidents do occur making varying levels of impact to our business. One of the many steps involved in the aftermath of the incident is measuring its impact.

Accurate measurement of financial impact due to an incident is an important part of incident management and is needed in real-time for many reasons including regulatory requirements.

However, it's not easy to calculate the impact given the dynamics of the complex distributed system. As SRE is deep-rooted in automation, we have built an Incident Impact Calculation Framework that accurately measures the incident impact using various Statistical and ML Models. It runs in seconds, is completely automated and fits well with our other incident management tools.

Building a real-time, reliable, and robust impact assessment framework is not as straightforward as it seems. Join us in this session to know more about impact assessment, its design, and challenges.

Prometheus is the leading open-source metrics monitoring solution. Prometheus 2.x contains a highly optimised Time Series Database (TSDB). It has ACID compliance and works efficiently with 100s of millions of time series on a single node. On top of its code base, Cortex, Thanos and Mimir have implemented distributed TSDBs.

The Prometheus TSDB has many components that make this all possible—an in-memory database, a Write-Ahead Log (WAL) for durability, memory-mapping of data from disk, persistent immutable data blocks with highly optimised inverted indices, and various techniques of maintaining this data in the background.

This talk will take you through the entire lifecycle of a time series sample (timestamp int64, value float64) starting from the point it enters the Prometheus TSDB's in-memory database until it gets deleted from the persistent blocks according to the retention policies.

Implementing observability was a game-changer. We dramatically reduced our time to identify problems, isolate causes, and see effects of changes.

But it's not quite as easy to retrofit as we might like to think. Brooks taught us to be wary of doing things over, but we couldn't safely make even basic changes to the existing codebase. Being able to do observability at all was a major motivation for a massive re-engineering. We'll share lessons learned as we rebuilt a large distributed system.

As we iterate the code we iterate our telemetry, too. Once you've learned something and changed the system, it's a new system; telemetry is not a continuous function! This has a drawback: you can't use observability as a substitute for business metrics. Which raises an interesting question: can you actually measure your SLOs using SLIs in a distributed system?

As an SRE or an application owner, it is common to come across the below questions/scenarios during the day-to-day activities of an engineer:

• "If only we had seen this sooner…" during the course of a SITE incident.
• "What happens if one of my service dependencies fails?"
• "How reliable my application is in the production environment?"

Chaos engineering has evolved into a must-to-have SRE culture that addresses the above questions and thereby improves the resiliency of internal systems that gives the teams confidence and a path to provide best-in-class products at scale.

In this talk, we will cover

1. The Chaos principles
2. How to prepare for Chaos journey in an organization
3. How to conduct Chaos Gamedays
4. How to Measure and Track the resiliency of a system
5. Leverage existing opensource Chaos platforms

As SREs who offer MySQL as a service to developers, we were presented with the challenge of providing support when applications hit connection limits or performance bottlenecks due to incorrect client side configurations or other resource constraints. To solve these operational problems, we developed a set of custom wrappers over open-source and native MySQL client libraries to provide observability into the client drivers and pools, in order to be able to quickly identify issues and mitigate them. In this we talk, we discuss the following

• Benefits of introducing observability into database clients
• Key metrics to monitor on application clients of databases
• Improving service availability with client side intelligence

Further, we speak on how we improved security of our services by developing a custom server plugin for MySQL that performs authentication using mTLS.

Modern services are equipped with hundreds of tunables. There are a lot of these tunables such as worker pool sizes, autoscaling policies, throttlers and circuit breakers that directly effect the service resilience. Finding ideal initial values for these tunables requires deep technical expertise. Also, these workloads change over time, requiring regular effort to re-tune stale parameters. As a consequence, configuration errors have become a source of operational toil and one of the major causes of overload, cascading service and system failures across the industry. Services should aim to expose a minimal configuration surface by dynamically adjusting parameters based on observations. Praveen will provide a deep-dive into how CrowdStrike is using real-time Adaptive Controls(inspired from TCP congestion control) to dynamically tune these parameters for improved resiliency using real-time sampling of errors and latencies, removing the need for periodic adjustment. He will also discuss lessons learned deploying the feature to CrowdStrike's massive production systems that handles multiple trillions of events per day without causing any incidents.

Imagine you run an automated ice cream gigafactory. You notice your vanilla ice cream output has dropped. You check your monitoring system—it can't find the problem because the error is hidden somewhere in your thousands of ice cream machines (you make too much ice cream).

After days of manually searching the factory:

1. You find a pond of vanilla ice cream next to a row of faulty machines, and
2. You realise that 30% of your ice cream melts on its way to be packaged.

In cloud SaaS products (no, not Sugar as a Service), good system monitoring is similarly crucial. We'll discuss how we monitor Jira at Atlassian for millions of customers, and explore some challenges we've faced. We'll go over best practices and solutions to help you maintain the sweetest systems, with a particular focus on managing big, high-cardinality data.