LISA '07 – Abstract
Pp. 63–78 of the Proceedings
Awarded Best Paper!
Application Buffer-Cache Management for Performance: Running the World’s Largest MRTG
David Plonka, Archit Gupta, and Dale Carder, University of Wisconsin—Madison
Abstract
An operating system's readahead and buffer-cache behaviors can
significantly impact application performance; most often these better
performance, but occasionally they worsen it. To avoid unintended I/O
latencies, many database systems sidestep these OS features by
minimizing or eliminating application file I/O. However, network
traffic measurement applications are commonly built instead atop a
high-performance file-based database: the Round Robin Database (RRD)
Tool. While RRD is successful, experience has led the network
operations community to believe that its scalability is limited to
tens of thousands of, or perhaps one hundred thousand, RRD files on a
single system, keeping it from being used to measure the largest
managed networks today. We identify the bottleneck responsible for
that experience and present two approaches to overcome it.
In this paper, we provide a method and tools to expose the
readahead and buffer-cache behaviors that are otherwise hidden from
the user. We apply our method to a very large network traffic
measurement system that experiences scalability problems and determine
the performance bottleneck to be unnecessary disk reads, and page
faults, due to the default readahead behavior. We develop both a
simulation and an analytical model of the performance-limiting page
fault rate for RRD file updates. We develop and evaluate two
approaches that alleviate this problem: application advice to
disable readahead and application-level caching. We demonstrate
their effectiveness by configuring and operating the world's
largest Multi-Router Traffic Grapher
(MRTG), with approximately 320,000 RRD files, and over half a million
data points measured every five minutes. Conservatively, our
techniques approximately triple the capacity of very large MRTG and
other RRD-based measurement systems.
- View the full text of this paper in HTML and PDF.
Listen to the presentation in
MP3 format.
The Proceedings are published as a collective work, © 2007 by the USENIX Association. All Rights Reserved. Rights to individual papers remain with the author or the author's employer. Permission is granted for the noncommercial reproduction of the complete work for educational or research purposes. USENIX acknowledges all trademarks within this paper.
|
