NVMe goes mainstream

Until last month, the ultra-fast, low latency, Non-Volatile Memory Express (NVMe) interconnect, was limited to a few fabrics and direct attach. But with NVMe/TCP, NVMe can now run over mainstream Ethernet networks. Here's what you need to know.

NVMe runs over the PCIe interconnect, so in its original form it is fast and has low latency. It was developed in response to the widespread use of SSDs when it became obvious that existing software stacks and interconnects - SATA, SAS - added too much latency and not enough bandwidth.

NVMexpress.org is the keeper of the NVMe specs, and they realized years ago that straight PCIe isn't very scalable, although some PCIe switches are now available. NVMe-oF (over Fabric) answered that objection and previously supported Infiniband, popular with supercomputers, and Fibre Channel, still widely deployed in enterprise data centers.

But the mainstream of innovation these days is in the cloud and other scale-out infrastructures, where Ethernet reigns supreme. And now that 100Gb/sec Ethernet is percolating into the market, it also has the most bandwidth as well.

Why not RDMA?

That was my first question as well, but there's a good answer. RDMA (Remote Direct Memory Access) over Ethernet requires setting up the network for lossless transmission, which is a bit of a pain, but it may offer lower latency. If need be you can run both on the same network.

NVMe/TCP doesn't require a lossless network. Nor does it require special hardware, although it is reasonable to assume that some hardware assists will be offered as it gains market traction.

It does require software. NVMe/TCP host software is available in Linux Kernel v5.0. There's also target software available for storage developers.

The Storage Bits take 

2020 will be NVMe's breakout year. PCIe v5 - 32GT/s - is coming out this year, and NVMe/TCP should be fully specified this year as well.

Up until now we haven't been able to take full advantage of SSD performance, due to protocols and interconnects that weren't as fast as what rack scale SSDs are capable of. That's about to change.

With NVMe/TCP, NVMe is the first storage protocol that runs on everything from mobile devices to the largest cloud data centers. As CPU performance progress drops, that's a very good thing for anyone who relies on data storage, either locally or in the cloud.

Comments welcome!