SREcon18 Asia/Australia Conference Program

Monitoring a.k.a figuring out what production code is doing is extremely important for an SRE organization. Monitoring services the right way can have a profound impact on how we do SRE. Modern software systems can be incredibly complex, code running on thousands of machines, depending on services we don't control and running code on user devices. Observing behavior of such systems means we have to change how we think about monitoring.

This talk will go over what a modern monitoring infrastructure for running software at scale looks like:

• Asking the right questions—how to decide what to monitor
• Types of data we want to collect and what answers it can help us find
• A look at how we build services at Facebook
• Collecting, storing and querying monitoring data at scale
• When things go wrong—what makes for a good alarm and what makes a bad one
• Putting it all together—debugging an outage using data

As an attendee, you will come out of the talk with fresh ideas about logging and monitoring. You will hear how we tackle these problems at Facebook, and why we do things the way we do.

Many companies want to grow a site reliability engineering team, but first need to ask “Is my company ready for SRE?” Taking the "lid-off" one of the most mature SRE organizations, I’ll describe the cultural tenets that provide the foundation for a high-trust and inclusive environment that is necessary for any SRE team to exist and evolve.

Every SRE team attempting to manage, mitigate or eliminate the risks facing their system will encounter two fundamental problems:

1. As humans our intuitive judgement about risk is unreliable.
2. The work required to address all potential risks far outstrips our available time and resources.

The CRE team (Customer Reliability Engineering—a group of Google SREs who partner with cloud customers to implement SRE practices in their application and across the cloud provider/customer relationship) battles these challenges every day in our interactions with customers. We have drawn on Google’s deep experience managing reliable systems, and the broader field of risk management techniques to develop a process that allows us to communicate an objective ranking of risks and their expected cost to a system. This ranking and the associated cost data can then be used as an input to team and business decision making.

This talk will cover the development of our process, explain how anyone can apply it to any system today and demonstrate how the resulting ranking and costs provide objective, consistent data which can take the tension and subjectivity out of often tense discussions around work priorities and focus (e.g. more features or more reliability?).

SRE is often perceived as an emergency response function—dealing with incidents and restoring system health. While it is true that there is much useful work done in this space, reactive processes impact only the MTTR not the MTBF. Once the low-hanging fruit of detection and remediation improvements are gone, improvement takes more and more investment. At this point, I'd argue, it is time to start preventative work.

We saw some reduction in incident rates through establishing a Post-Mortem process but these often involved only the poor souls who happened to have been called upon to help triage and fix the issue. Realisation dawned on me that attempting to evangelise to engineers with little influence on the balance of functional/non-functional development effort was going to have limited success.

By embarking on a campaign evangelising reliability (similar to the way forest fire prevention, or health promotion campaigns might work) and targeting the right level within the organisation, we're seeing positive cultural and behavioural changes, and better operational morale and we believe it will result in fewer severity outages.

When you first built services in one data center, it is always easy to do the deployments and changes. When the number of services increases, you would spend more time on service management and governance.

When we package all the services into one cloud offering and need to deploy in hundred of data centers, we have new problems. We need to evaluate all the capacity and data center or even network requirements of each services, and then initialize the new data center based on the evaluation. Normally it would need experts from datacenter, network and software development team with one or two months. How to do all these things automatically only based on the capacity planning artifacts? In this talk, I’ll share some of how we build and ship Apsara Stack, a dedicated cloud offering from Alibaba, to the data centers from different vendors.

Specifically:

1. A full datacenter capacity planning based on the measures from users' perspective.
2. A unified deploy model for network, operating systems and various cloud products.
3. Maintaining lightweight, easy-to-change production configuration instead of providing different toolsets.
4. Validate and test all changes automatically when changing any deployments

In this talk, I will discuss the different aspects of incident management and how we've built automation around each part at Xero. This includes: transforming manual alerts into automatic notifications through an issue report pipeline; a chat bot that streamlines incident coordination by facilitating effective communication and providing guidance through the process; and how we extract data from each incident for postmortem review. I'll also discuss how our tools have evolved and the lessons we learned on the way.

Have you thought that your model trained on a Monday might not work on Saturday? Or that the model that you trained on users in Florida might not work for all Spanish-speaking users? In this talk, we present lessons learned from deploying and productionizing ML systems across various products at Google.

Monitoring system is vital for service stability and availability. To support Baidu’s massive services and machines, the metrics being collected has grown to 1 billion. These metrics must be stored in a reliable and efficient database, which must support real-time insertion of new data and various queries, ranging from aggregation, alerting, to reports and visualization, with diversified time granularities.

Our time-series database (TSDB) consists of three layers, a memory database based on Redis stores hot data, a HBase stores warm data, and a HDFS stores cold data. To achieve efficient insertion, we extensively apply batch and asynchronous methods to the write path, in addition to HBase’s ability of high throughput writing. To improve reading performance, we design specialized data model and embed multi-layer down-sampling mechanism into HBase. The memory database incorporates compression techniques to serve real-time, frequent, and small queries, while preserving memory consumption at a reasonable level. All the data are backed up in a separate HDFS to support offline analysis.

In this talk, we will explore the challenges of large scale time-series processing, and introduce our practice of building TSDB. We will also share some successful experiences, such as retention policy, and trade-offs between cost and performance.

When it comes to high availability (HA) or user experience (UE), people often think about the stability of backend services or product design. Network connectivity, especially the Internet connectivity, is neglected. This might partly come from the impression that network is usually stable. Our observation shows that network failures are far from scarce, at least in China. Every week, we detect 3-5 PoP failures, 5-10 backbone failures breaking the connectivity at the province level. Most of the failures can be remedied by modifying DNS setting to bypass the broken path. The remediation depends on two systems, the detection system and the traffic scheduling system.

The detection system must detect failures precisely and punctually. Besides dedicated monitoring agents, we recruit volunteering agents to improve coverage and punctually. Dealing with their heterogeneity and unpredictable presence is crucial to the detection performance.

The traffic scheduling system is responsible for detouring the traffic to the correct path. It must consider not only the connectivity of external network links, but also the users’ experience and the load of target IDCs.

In this talk, we will introduce how to implement and use the above two systems to handle Internet connectivity failures.