Bobtail: Avoiding Long Tails in the Cloud

by George Porter

Shared cloud infrastructures provide application developers with numerous advantages, including the ability to seamlessly scale up or down resources in response to demand, and the ability to focus on developing their applications, without having to purchase, manage, or wait for dedicated data center resources. However, these advantages come at a cost, namely that the developer can no longer reason about dedicated compute, storage, and networking resources. This paper addresses a particularly important and problematic instance of this problem, namely high "tail latency,'' or the 99.9th percentile of service response time. For distributed applications that compose numerous network requests into a single, logical operation, tail latency is particularly important, since it often determines the overall response time of the application as a whole. Designing performant applications requires minimizing the latency variance of individual network operations, which is a challenge in shared cloud environments.

In this paper, the authors have made two primary contributions. First, they have carried out a multi-week long measurement study on the popular Amazon EC2 cloud platform to determine the source of tail latency. The study found that it is not the network topology or congestion that is to blame, but rather the node, and in particular the virtual machine scheduler. When latency-sensitive and CPU-intensive VMs are co-scheduled together, that result is high tail latency. The program committee was particularly impressed with thecare that went into this measurement study, and how through the study the authors were able to determine that a simple node-local test (measuring the accuracy of timer events) predicts the expected latency of network operations. While it has generally been widely known that shared cloud environments exhibit higher tail latency than dedicated data centers, the program committee was pleased to see a detailed, quantitative analysis of the specific sources of that latency and their effect on observed application performance.

The resulting node-local test forms the basis of this paper's second major contribution, namely the Bobtail system. With Bobtail, cloud users are able to select a subset of their nodes that are likely to exhibit low tail latency, and do so without support from the cloud provider. This is critical for developers that target black-box cloud infrastructure or spread their applications across heterogeneous cloud providers. One concern raised by the program committee, and echoed by the authors themselves, is avoiding a "race to the bottom'' scenario where multiple users relying on Bobtail pursue the same set of resources, reducing the overall utility of the infrastructure. The authors point out that their findings show that it is not VM co-scheduling that results in high tail latency, but specifically co-scheduling latency-sensitive and CPU-intensive workloads together. The authors' commendable efforts in measuring EC2 do not necessarily carry over to other cloud providers; however, their findings show that the primary sources of tail latency are hypervisor-specific, meaning that Bobtail is likely applicable to any cloud relying on Xen. Further experience with other platforms might bear out this hypothesis.
In summary, providing users with tools to better reason about the properties of shared cloud infrastructures is critical to their adoption, and Bobtail is a step in that direction.