Flash storage and the diseconomies of sharing

One little appreciated aspect of the flash storage revolution is that it has dramatically changed the economic relationship between individual SSDs and all-flash-arrays (AFAs), compared to disk drives and disk-based arrays. How?

Disks topped out at about 500 IOPS (Input/output operations per second), while disk-based arrays could regularly manage 50,000 to 100,000 IOPS, and the biggest arrays could manage millions of IOPS. To be conservative, call the array to device IOPS ration a 1,000 to 1, but it could go as high as 10,000 to 1.

That gap is so large that there is no way that an individual disk, or 10 of them, could substitute for a competently configured array. And they never did, which is one reason why the SAN (storage area network) market took off 20 years ago.

The other reason the SAN market took off is that the services bundled into the array controller made the array more useful in a shared environment than individual disks inside servers could be. But this required a hefty CPU and lots of firmware to add value to a box of disks. And THAT meant that the disk cost of a storage array was a fraction of the array's total price.

That was then

Today the performance of individual SSDs is as fast as many AFAs for a fraction of the price. For example the Dell PowerEdge Express Flash NVMe Performance PCIe SSD is spec'd at ≈700,000 IOPS. For a $2-3K card to perform in the neighborhood of a $100,000 AFA is a stunning 1 to 1 ratio.

So that leads to the diseconomies of flash: do you invest in a single AFA for $100k and share it among 10 servers, or do you buy 10 PCIe/NVMe SSDs and put one in each server? If raw performance is the only issue, the answer is obvious.

But there's a wrinkle: what if your servers need to share data, such as multiple databases? Maybe with extra vigilance and some smart front end software you can shard the databases across 10 servers and save $80k on hardware. But extra vigilance and smart front ends aren't free either. It's likely that your go fast SSDs will end up only using 15-25 percent of their performance, and you'll be forever tuning everything to keep the wheels from falling off.

The Storage Bits take

The SSD guys know this, and they don't know what to do. Ten years ago, David Flynn, one of the founders of Fusion-io and a supercomputing veteran, wanted to enable his very fast SSDs to be shared across a network. But the problem was too big for the company.

But whenever the economic payoff is this big, someone thinks of something. For years, purists insisted that SCSI drives were essential for storage arrays. But the cost advantage of mass-produced SATA drives led people to fix the problems and made them common.

So get ready for a solution that enables high-performance sharing of server SSDs. It's too juicy a target to ignore.

In the meantime, you're stuck. If you need to share data across multiple servers, then you need a SAN and an AFA or a more affordable hybrid array. But someone will figure this out, and hopefully soon.

Courteous comments welcome, of course.

Hybrid cloud storage, first look: Synology DS916+ super-NAS: