Disk performance hits the wall

Disk drives have come a long way from the massive rotating plates of the 1950s and 1960s, when 200MB was a huge volume, to today when disk drives containing 2TB and up are considered commonplace.

But this massive improvement has a little-considered dark side. It's not so much a problem for desktops but for an enterprise, the exponential growth in capacity generates huge performance problems. Speed simply hasn't increased at anything like the same rate, as this fascinating paper from IBM points out.

The reason this is a problem is that doing anything with the huge amounts of data that enterprises are now storing takes longer and longer. We're talking about tasks such as database indexing, backing up, data migrations -- anything in fact that involves large numbers of transactions.

That's because a disk that contains 2x as much data as that of three years ago is in no way close to twice as quick. In fact, IBM's paper, GPFS Scans 10 Billion Files in 43 Minutes, published by its advanced storage lab in San Jose, California, shows that latency has remained unchanged for at least five years, while the rest of the performance metrics are improving at modest rates of between five and 20 per cent annually. They're not keeping pace with Moore's Law.

The traditional way of fixing that problem is to add more spindles, which boosts throughput but also adds capacity, often where it's not needed and not required -- and adding spindles also adds complexity, cost, heat and likelihood of failure.

SSDs would seem to be the way out of this as they clearly don't suffer from the same problems but the file system is also a crucial component. With the growth of object storage and the like, this is just the start of a very interesting technological voyage. With the availability now of high-end file systems such as ZFS for free, it might not be too long before end-users can all enjoy the benefits too.