Beyond Repair: Proactive Maintenance Work at Scale

FBAR has enabled Facebook's production engineering teams to automate break/fix responses to many of the single-host events, such as hardware failures or application daemon crashes, that occur on a daily basis, leaving engineers to focus on larger, more complex and interesting problems.

To provide more opportunities and time for engineers to take on more meaningful work, the team developed automated responses for the less frequent, but larger-scale proactive work necessary to maintain a healthy infrastructure. This work takes on different shapes, including top-of-rack switch replacements, disruptive BIOS/firmware updates, or work on power supply (backup or primary).

This talk will use a (fictitious) example of a set of racks undergoing maintenance to give an overview of how the automation provided by FBAR to handle single-host repairs was expanded to cover proactive maintenance work, including an explanation of the automated maintenance process, the API engineers implement to automate the work, and the way it interfaces with humans when automation won't (or fails to) work. It will also include a few small lessons we learned along the way, and explain why a simple approach covers much of the use cases with few drawbacks.

Romain is a manager in Facebook's Production Engineering organization, working on the team that maintains Facebook's Auto Remediation tool (FBAR). He originally came in to Facebook as part of the Site Reliability Operations team and has spent the last five years focused on safely automating operational tasks. Most recently, this has taken the shape of creating new tooling that allows us to automate maintenance affecting multiple machines (and multiple racks) at a time. The team has spent the last two years refining the process and keeping automation simple while covering the majority of use cases.