Mastering Linux Performance Tools

All kinds of applications run on Linux, from web servers to distributed database engines and embedded applications. Troubleshooting performance in the field, especially when invasive profilers can't be used, is a delicate art that requires a solid understanding of the system and low-overhead tools. In this workshop, we will visit a spectrum of Linux performance monitoring tools.

We will start with a simple performance checklist based on the USE method, including tools like top, iostat, vmstat, mpstat, sar, and others. Then, once we identify the overloaded resource, we will dig in deeper using perf: tracepoints, hardware events, dynamic probes, and USDT. We will also collect stack traces of heavy events (CPU usage, disk accesses, network) and visualize them using flame graphs.

Finally, we will discuss the emerging superpower for Linux performance monitoring: BPF and BCC. This is a new kernel technology that enables low-overhead, super-efficient monitoring and tracing tools, which perform aggregation closer to the source where the events occur and provide useful information at a fraction of the cost. We will review a performance checklist based on BCC tools, and explore one-liners from the general-purpose trace and argdist tools.