dsync: Efficient Block-wise Synchronization of Multi-Gigabyte Binary Data

Backing up important data is an essential task for system administrators to protect against all kinds of failures. However, traditional tools like rsync exhibit poor performance in the face of today's typical data sizes of hundreds of gigabytes. We address the problem of efficient, periodic, multi-gigabyte state synchronization. In contrast to approaches like rsync which determine changes after the fact, our approach tracks modifications online. Tracking obviates the need for expensive checksum computations to determine changes. We track modification at the block-level which allows us to implement a very efficient delta-synchronization scheme. The block-level modification tracking is implemented as an extension to a recent (3.2.35) Linux kernel.

With our approach, named dsync, we can improve upon existing systems in several key aspects: disk I/O, cache pollution, and CPU utilization. Compared to traditional checksum-based synchronization methods dsync decreases synchronization time by up to two orders of magnitude. Benchmarks with synthetic and real-world workloads demonstrate the effectiveness of dsync.