libvfn: A Low-level NVMe Application and VFIO Driver Framework

This talk presents the design and implementation of libvfn, a new open-source library for interacting with PCIe-based NVMe devices from user-space using VFIO. The core of the library is excessively low-level and aims to allow NVMe controller verification and testing teams to interact with devices at the register and queue level. While the library ships with a production ready NVMe driver with a high-level API, the library is designed to expose enough low-level VFIO functionality with which custom drivers can be implemented for any PCIe device as required by the application. We begin with a brief introduction to PCIe and NVMe in the context of VFIO-based user-space drivers, followed by a technical discussion on the implementation of the included NVMe driver. We discuss how the design and API differentiates itself from the current state of the art (i.e. the Storage Performance Development Kit) and how it enables interaction with advanced features of NVMe at the PCIe level. We end the technical part by showing how libvfn has been successfully integrated into existing projects with a case study on the "vfio" backend for the open-source xNVMe project. Finally, we wrap up by discussing performance and show that libvfn is on par with that of SPDK.