Object storage inches closer to mainstream

I may have mentioned here before that I believe object storage is likely to become more popular, and here's why.

Standard file systems tend to slow down the more files they have to manage due to the need to traverse the tree structure, which can be very deep for service providers managing data from multiple customers. And as the index tends to grow with file numbers, retrieving it from disk can take too long when performing simultaneous lookups for hundreds or thousands of users. The result is that end users experience inordinately long file retrieval times.

This problem is growing as the volumes of stuff being stored grow, and -- to make matters worse -- storage systems consolidate in service providers' datacentres. In other words, there are more and bigger files, and they're being piled up in fewer places rather than distributed around the network.

The answer offered by the storage industry is object storage. In a nutshell, most object storage systems have at least a couple of the following attributes in common:

1. A ring or cluster of file servers to manage the file system 2. A file system consisting not of a hierarchical tree-like structure into which files can be dropped, willy-nilly, but a set of identically-sized container files in a flat, single directory 3. User files are held as objects in the containers 4. A huge index, probably held in RAM, that points to the locations of individual files 5. Retrieval of file locations is measured in microseconds irrespective of their number

Such file systems are designed to manage storage systems containing very large numbers of unstructured files -- end user files, videos, emails and the like, rather than databases. They're more scalable, say object storage vendors, because you can just add more servers when you need to manage more files, and more resilient because each server knows what data their companions contain and can take over should one disappear from the ring. Users will tend to be large enterprises, and service and cloud providers managing file numbers into the billions.

Is this stuff for real? Red Hat just bought Gluster (it stands for GNU cluster) for $136 million, which puts Red Hat in the high-volume storage business. It makes sense when you consider that many of RH's customers for its JBoss middleware products are large service providers and enterprises.

I suspect you can expect to see further acquisitions of companies such as Scality, Caringo and Amplidata as the big players find a need for higher levels of scalability. The exception to this is Dell, which already has an object file system offering. Now there's a thing.