Want to Solve Over-Monitoring and Alert Fatigue? Create the Right Incentives!

Telemetry monitors and their (constant) beeping is a pretty common sight in hospitals. I saw these at the NICU where my twins were being cared for after being born prematurely. My wife and I used to freak out every time one of these went off. Unlike a missed alarm that said your site's down, failing to act on an alarm at a hospital can have much more critical consequences; in 2010 at a hospital in Massachusetts, a patient's death was directly linked to telemetry monitoring after alarms signaling a critical event went unnoticed by 10 nurses.

I attempted to solve this problem when I joined Zynga (in 2013) as the head of SRE. I will go over our failed attempts including filtering the noise, adding heads, building more tools, etc. Will also cover how I came up with an initiative called "clean room" as a way to incentivize engineering teams to keep the noise levels low. Finally, go over some of the tactics that worked (and ones that didn't).

Most people I spoke to about "clean room" almost always walked away having learned something (some have said it's common sense). Share, learn, ask questions, participate - I'll try to make it fun!