Clustered and Parallel Storage System Technologies

This tutorial is oriented toward administrators and developers who manage and use HPC systems, and especially for those involved with storage systems in these environments. Storage is often a critical part of the HPC infrastructure. An important goal of the tutorial is to give the audience the foundation for effectively comparing different storage system options, as well as a better understanding of the systems they already have.  

Cluster-based parallel storage technologies are used to manage millions of files, thousands of concurrent jobs, and performance that scales from 10s to 100s of GB/sec. This tutorial will examine current state-of-the-art high-performance file systems and the underlying technologies employed to deliver scalable performance across a range of scientific and industrial applications.

The tutorial starts with a look at storage devices and SSDs, in particular, which are growing in importance in all storage systems. Next we look at how a file system is put together, comparing and contrasting SAN file systems, scale-out NAS, and object-based parallel file system architectures.

Topics include scaling the data path, scaling metadata, fault tolerance, and manageability. Specific systems are discussed, including Lustre, GPFS, PanFS, HDFS (Hadoop File System), OpenStack, and the NFSv4.1 standard for parallel I/O. We continue up the stack to discuss MPI-IO middleware that is often used in large parallel programming environments for efficient I/O at scale.