Vault '20 Conference Program

The global hunger for more storage capacity requires us to rethink conventional storage interfaces that we use in today's storage infrastructures. The Zoned Namespaces (ZNS) SSD interface is a new standardized storage interface, which is being adopted by over 31% of the total cloud market. When compared to typical enterprise SSDs, ZNS SSDs get more than 20% additional capacity, require an order of magnitude less device DRAM while improving its tail latency (QoS). The Zoned Namespace interface defines sequential write required zones (similarly to ZAC/ZBC for SMR HDDs), which requires that host applications must write sequentially. This talk focuses on the newly introduced Zone Append I/O command, which enables a ZNS SSD to perform data placement within a given zone. The Zone Append command enables new ways to innovate in your storage system. Such as (1) writing directly from clients to ZNS SSDs when using a distributed file-system, (2) offload of fine-grained data placement to the SSD, and (3) eliminate single writer per zone bottleneck.

Due to travel restrictions, Naohiro Aota could not attend the conference and their work was presented by Johannes Thumshirn, Western Digital, Inc.

Zoned block device (ZBD) support has been introduced in Linux with kernel version 4.10. ZBDs have different write constraints than regular block devices. A ZBD is divided into several zones and each zone must be written sequentially and must be reset before rewriting.

The main type of ZBD currently available is SMR HDDs. The NVMe Zoned NameSpace proposal is also being drafted to add a zone abstraction to the NVMe specifications.

Natively supporting ZBDs in a filesystem is not a trivial change. Some filesystems must rely on special block layer drivers to ensure sequential writes (e.g. ext4 and the dm-zoned device mappers). Filesystems using a copy-on-write design are better candidates for native ZBD support. Examples are F2FS and btrfs.

This talk discusses the principles of ZBD and ZNS native support in filesystems. Support in F2FS is discussed and the approach taken with btrfs is next presented. This is followed with a performance comparison between filesystems with native ZBD/ZNS support and regular ones using dm-zoned.

The DAX functionality provides a mechanism to leverage DAX features from within existing filesystems. It is currently geared primarily for the 'mmap()' call, allowing applications to use the benefits for persistent memory while still having a normal filesystem. However, most filesystems are designed to handle efficient large block I/O, as with this traditional storage works best. Unfortunately, this is precisely what NVDIMM is notoriously bad at; NVDIMM really excels at small, even bit-wise, I/O patterns; in fact, some systems even have a performance penalty for large I/O blocks. Also, tests with standard I/O performance tools like 'fio' do not show any noticeable benefit from using DAX.

xadfs inverts this concept; use mmap() as the default I/O path, and map all other calls onto that. The prime goal here is to use NVDIMMs as efficiently as possible to demonstrate which performance benefits could be expected. With these results we will be gaining a better understanding for which use-cases NV-DIMMs should be employed, and which are better left to traditional storage.

In this talk I will be presenting a proof-of-concept implementation of xadfs, and providing a performance comparison with traditional filesystems implementing DAX functionality.

The popularity of NVMe has gone beyond the limits of the block device. Currently, NVMe is standardizing Key-Value (KV) and Zoned (ZNS) namespaces, and discussions on the standardization of computational storage namespaces have already started.

While modern I/O submission APIs are designed to support non-block submission (e.g., io_uring), these new interfaces incur an extra burden into applications, who now need to deal with memory constraints (e.g., barriers, DMA-able memory).

To address this problem, we have created xNVMe (pronounced cross-NVMe): a user-space library that provides a generic layer for memory allocations and I/O submission, and abstracts the underlying I/O engine (e.g., libaio, io_uring, SPDK).

In this talk, we (i) present the design and architecture of xNVMe, (ii) give examples of how applications can easily integrate with it and (iii) provide an evaluation of the overhead that it adds to the I/O path.

The Skyhook Data Management project (SkyhookDM.com) at the Center for Research in Open Source Software (cross.ucsc.edu) at UC Santa Cruz implements customized extensions through Ceph's object class interface that enables offloading database operations to the storage system. In our previous Vault '19 talk, we showed how SkyhookDM can transparently scale out databases. The SkyhookDM Ceph extensions are an example of our 'programmable storage' research efforts at UCSC, and can be accessed through commonly available external/foreign table database interfaces. Utilizing fast in-memory serialization libraries such as Google Flatbuffers and Apache Arrow, SkyhookDM currently implements common database functions such as SELECT, PROJECT, AGGREGATE, and indexing inside Ceph, along with lower-level data manipulations such as transforming data from row to column formats on RADOS servers.

In this talk, we will present three of our latest developments on the SkyhookDM project since Vault '19. First, SkyhookDM can be used to also offload operations of access libraries that support plugins for backends, such as HDF5 and its Virtual Object Layer. Second, in addition to row-oriented data format using Google's Flatbuffers, we have added support for column-oriented data formats using the Apache Arrow library within our Ceph extensions. Third, we added dynamic switching between row and column data formats within Ceph objects, a first step towards physical design management in storage systems, similar to physical design tuning in database systems.

Massive storage systems composed of tens of thousands of disks are increasingly common in high-performance computing data centers. With such an enormous number of components integrated within the storage system the probability for correlated failures across a large number of components becomes a critical concern in preventing data loss. To better protect against correlated failures we introduce Single-Overlap Declustered Parity (SODP), a novel declustered parity design that tolerates large numbers of disk failures and minimizes rebuild time. Our evaluation results show during large failure bursts SODP improves protection against data loss by 20x compared to traditional declustered parity and provides almost identical rebuild times compared to the current state of the art.

The Crimson project is an effort to build a replacement ceph-osd daemon well suited to the new reality of low latency, high throughput persistent memory and NVMe technologies. Built on the seastar C++ framework, crimson-osd aims to be able to fully exploit these devices by minimizing latency, cpu overhead, and cross-core communication. This talk will discuss the design, current status, and future direction of the crimson project.

Ceph is a unified distributed storage. Caching plays a significant role in enhancing performance in a distributed software. In this regard, caching in Ceph user-space has been evolving. This talk will explore the different caching policies that exist today for the Ceph block storage (RBD) along with going into the design details of upcoming feature work which is a write-back cache implementation on SSDs and persistent memory

Linux block integrity is a well-known block layer subsystem that helps to detect and prevent data corruption. This talk is based on hands on experience in building and testing storage systems and provides solutions for challenges faced in the block integrity stack. It also covers specifics of integrity implementation in SCSI or NVMe kernel drivers as well as in virtual environments: qemu, virtio, vhost.

Encrypting data at rest is a must-have for any modern SaaS company. And if you run your software stack on Linux, LUKS/dm-crypt is the usual go-to solution. However, as the storage becomes faster, the IO latency, introduced by dm-crypt becomes rather noticeable, especially on IO intensive workloads.

At first glance it may seem natural, because data encryption is considered an expensive operation. But most modern hardware (specifically x86 and arm64) platforms have hardware optimisations to make encryption fast and less CPU intensive. Nevertheless, even on such hardware transparent disk encryption performs quite poorly.

When looking into dm-crypt source code, we noticed that it has a lot of indirection and offloading: instead of encrypting/decrypting IO requests synchronously dm-crypt offloads every operation to a dedicated thread. By making a simple PoC patch and removing all the offloading code we were able to speed up the overall read speed from an encrypted block device by 200%-300% depending on the block size.

This talk aims to revisit the architecture and design choices of the dm-crypt module and research ideas on how to make Linux transparent disk encryption faster.

Development of Samba, the Open Source File/Print/Authentication/Active Directory server for Linux and FreeBSD is accelerating. Come and hear about the new developments in Samba, the most widely used File Server in the Open Source world, and increasingly used as a gateway to Cloud File Storage.

To implement the SMB protocol, Samba has to implement semantics that are not covered by the Linux kernel API. The protocol element to mention here are the concept of share modes and leases, similar to NFSv4 share reservations and delegations. To implement those, Samba has to maintain data structures in user space and keep those consistent across cluster nodes. One of those data structures is a central table containing SMB-level information about all file open instances.

This talk will describe the semantics to be implemented, the challenges for clustered implementations of the SMB protocol and approaches by the Samba Team to make this scale well across nodes.

Understanding the many Kubernetes storage ‘objects’ along with their not-always-obvious interaction and life-cycles can be daunting (Volumes, Persistent Volumes, Persistent Volume Claims, Volume Attachments, Storage Classes, Volume Snapshots, CSIDriver, CSINode, oh my...)

Perhaps the best ways to gleen a deep understand of these storage objects and how storage-related scheduling works in Kubernetes is to write a Container Storage Initiative (CSI) driver. While most of us will never need to write a CSI driver, in this session we will make storage with Kubernetes more accessible by exploring it from an inside-out approach learned by writing a CSI Driver.

In this session you will obtain an understanding of:

• How Kubernetes 'Volumes' relate to mounted storage available from within containers
• The Kubernetes Declarative Model
• The many Kubernetes Storage Objects
• Kubernetes Scheduling and how it is influenced by storage object
• How Kubernetes controllers move the storage objects through their life-cycles
• What is the role and responsibility of a storage type specific CSI Driver
• What are the roles and responsibilities of CSI support ‘Side-Cars’
• Putting it all together and relating this all back to how to write a CSI Driver