Improving Availability in Distributed Systems with Failure Informers

by Katerina Argyraki

This paper addresses one of the quintessential questions in system design: how much and what kind of information should applications get about lower-layer failures? Existing systems do not typically expose such information in a systematic way, leaving applications to detect failures themselves through end-to-end timeouts. The rationale behind this design choice was that the benefit applications would gain from explicit failure reports was not worth the cost of the mechanism needed to provide them. The paper argues that things have changed, and it is time our perspective on failure reporting did, too: these days, failures that reduce application availability can bear a significant financial cost; hence, we need to provide applications with the information they need to mitigate the impact of these failures as much as possible.

The first piece of the proposed solution is a "failure informer" interface for exposing host and network failures to applications. What this interface reports about each failure is (1) whether it has certainly occurred or is imminent and (2) whether it is certainlypermanent or not. This classification results in four failure types: a "stop" (the target process has stopped executing and lost its volatile state), an "unreachability" (the target process may still be running but the client cannot reach it), a "stop warning" (the target process may soon stop as it is running out of a critical resource) and an "unreachability warning" (the target process may soon become unreachable). The interface also reports the expected duration of the failure and, in case of warning, the critical resource responsible for the warning.

The second piece of the proposed solution is "Pigeon," a service that implements the failure-informer interface in the context of a single administrative domain running Open Shortest Path First (OSPF). Pigeon consists of several components running at end-hosts and routers: "sensors" detect faults (e.g., process exits, host or router reboots, link failures or overloads), "relays" communicate these faults to the interested clients, and an "interpreter" processes these faults and turns them into failure reports.

Does the failure-informer interface provide information that is necessary and sufficient for applications to respond to lower-layer failures in the best possible way? Is the benefit for applications worth the complexity of embedding sensors and relays in every end-host and router? These questions are impossible to answer without real deployment experience—or, at the very least, real-world failure data, which is rarely available to researchers. But the authors have done the next best thing: They have tested Pigeon with three real applications (running in a small number of virtual machines interconnected in a fat tree). Their evaluation shows that Pigeon helps a storage system avoid unnecessary failovers, a key-value store recover from a link failure several seconds faster, and a lease-based replication system reclaim the lease held by a crashed process severalseconds earlier. This is promising evidence that explicit failure reports can significantly improve application availability.