Figure 9 compares the performance of the five different mirroring solutions. The x-axis shows loss probability on the wide-area link being increased from 0% to 1%, while the y-axis shows the throughput achieved by each of these mirroring solutions. All mirroring solutions use 64 testers over eight storage servers.
At 0% loss we see that the local-sync and remote-sync solutions achieve the highest throughput because they do not use proactive redundancy, thus the goodput of the wide-area link is not reduced by the overhead of any forward error correcting packets. On the other hand, local-sync+FEC, remote-sync+FEC, and network-sync achieve lower throughput because the forward error correcting packets reduce the goodput in these cases. The forward error correction overhead is tunable; increasing FEC overhead often increases transmission reliability but reduces throughput. There is a slight degradation of performance for network-sync since SMFS waits for feedback from the egress router instead of responding immediately after the local kernel buffers the send request. Finally, the remote-sync and remote-sync+FEC achieve comparable performance to all the other configurations since there is no loss on the wide-area link and the storage servers can saturate the link with overlapping mirroring requests.
At higher loss rates, 0.1%, 0.5%, and 1%, we see that any solution that uses proactive redundancy (local-sync+FEC, remote-sync+FEC, and network-sync) achieves more than an order of magnitude higher throughput over any solution that does not. This illustrates the power of proactive redundancy, which makes it possible for these solutions to recover from lost packets at the remote mirror using locally-available data. Further, we observe that these proactive redundancy solutions perform comparably in both asynchronous and synchronous modes: in these experiments, the wide-area network is the bottleneck since overlapping operations can saturate the wide-area link.
Figure 10 shows the system throughput of the network-sync solution as the wide-area one-way link latency increases from 25 ms to 100 ms. It demonstrates that the network-sync solution (or any solution that uses proactive redundancy) can effectively mask latency and loss of a wide-area link.