Network Fault Finding System: Packet Loss Triangulation

Most network monitoring relies upon the devices providing the signal used to calculate health via syslog messages, SNMP, telemetry, or custom APIs.

In large scale networks, we can’t trust the devices to accurately report health in all the possible failure cases that may exist. At FB, in addition to standard monitoring tools, we also actively probe our networks with test traffic to ensure the platforms are behaving as we expect. With active monitoring, we can find misbehaving network elements even when they exist several layers deep inside the network.

During the presentation, we will show how we built a sample system that achieves similar results using open source tools and perform a live demo with a lab network from start to finish, introducing packet loss and showing how the system can identify where the loss is occurring in real time.