File Access Interfaces

Our highest bandwidths and lowest overheads are achieved using the file mapping mmap system call, rather than the traditional read/write interface. Differences among these interfaces have a negligible effect on delivered bandwidth at disk speeds, but the effect is substantial at gigabit-per-second network speeds. This is true even on modern platforms whose memory systems have sufficient bandwidth to serve the CPU and the I/O system simultaneously (e.g., Alpha Miata or Intel Pentium-II/440LX). The effect can be dramatic on platforms with lower memory system bandwidth (e.g., Alcor).

The experiments use three different file access schemes. The stream option uses read and write system calls. The mmap experiments use a single mmap system call to map the entire file into virtual memory, avoiding the copying inherent in the read and write interface. The memory-mapped block (MMB) experiments use a hybrid scheme that combines the benefits of the stream and whole-file mmap access policies. MMB is an an attractive interface for high-volume file access when performance is important and the uniform addressing of whole-file mmap is not required.

MMB uses the mmap system call in a block-oriented fashion, repeatedly mapping N regions of virtual memory of size B to different ranges of offsets in the file. Our MMB experiments use B=256K and N=2 (double buffering). Like whole-file mmap, MMB is a zero-copy file access scheme; it incurs slightly higher system call overhead than mmap, but our results show that this is insignificant with sufficiently low Nand sufficiently large B, and is overshadowed by other benefits. Like stream, MMB allows the application to explicitly specify its accesses to the kernel. This information can be exploited by the kernel to improve performance for applications that use MMB. Moreover, MMB is an asynchronous interface, allowing the application to specify accesses early in order to overlap file access with computation.

We have modified our Digital Unix kernels to detect MMB accesses and respond with the following policies:

• When an MMB mmap call is issued, the kernel immediately initiates an asynchronous prefetch of all pages in the newly mapped region, superseding the usual read-ahead policy.

• The memory frames that previously backed the remapped region are released, discarding the mapped-over file blocks from the local file cache, and pushing them to the network if necessary. This allows the application to control the local cache replacement policy by selecting the region to remap, and reduces the overhead of the paging daemon and LRU eviction code.

• We have extended the mmap interface with a MAP_OVERWRITE flag that allows the application to specify that the mapped file region will be overwritten without reading it; in this case, the kernel may simply leave the existing frames mapped, but change their identity in the file cache. The new flag fixes an inherent flaw in mmap: when a process references a page in a mapped region, the kernel does not know if the process will read from that page, so it must initialize the frame by zeroing it or reading it from the file. Our approach will expose corrupt data to a process that mistakenly reads from a region mapped with MAP_OVERWRITE, but it will never violate security by leaking data belonging to another process.

We emphasize that with the exception of the new flag, these policies do not change the semantics of the standard mmap interface, but only its performance. We leave a detailed study of the MMB interface and its usage to future work.