Object storage inches closer to mainstream

Object storage inches closer to mainstream

Summary: I may have mentioned here before that I believe object storage is likely to become more popular, and here's why.Standard file systems tend to slow down the more files they have to manage due to the need to traverse the tree structure, which can be very deep for service providers managing data from multiple customers.

SHARE:
TOPICS: Networking
5

I may have mentioned here before that I believe object storage is likely to become more popular, and here's why.

Standard file systems tend to slow down the more files they have to manage due to the need to traverse the tree structure, which can be very deep for service providers managing data from multiple customers. And as the index tends to grow with file numbers, retrieving it from disk can take too long when performing simultaneous lookups for hundreds or thousands of users. The result is that end users experience inordinately long file retrieval times.

This problem is growing as the volumes of stuff being stored grow, and -- to make matters worse -- storage systems consolidate in service providers' datacentres. In other words, there are more and bigger files, and they're being piled up in fewer places rather than distributed around the network.

The answer offered by the storage industry is object storage. In a nutshell, most object storage systems have at least a couple of the following attributes in common:

1. A ring or cluster of file servers to manage the file system 2. A file system consisting not of a hierarchical tree-like structure into which files can be dropped, willy-nilly, but a set of identically-sized container files in a flat, single directory 3. User files are held as objects in the containers 4. A huge index, probably held in RAM, that points to the locations of individual files 5. Retrieval of file locations is measured in microseconds irrespective of their number

Such file systems are designed to manage storage systems containing very large numbers of unstructured files -- end user files, videos, emails and the like, rather than databases. They're more scalable, say object storage vendors, because you can just add more servers when you need to manage more files, and more resilient because each server knows what data their companions contain and can take over should one disappear from the ring. Users will tend to be large enterprises, and service and cloud providers managing file numbers into the billions.

Is this stuff for real? Red Hat just bought Gluster (it stands for GNU cluster) for $136 million, which puts Red Hat in the high-volume storage business. It makes sense when you consider that many of RH's customers for its JBoss middleware products are large service providers and enterprises.

I suspect you can expect to see further acquisitions of companies such as Scality, Caringo and Amplidata as the big players find a need for higher levels of scalability. The exception to this is Dell, which already has an object file system offering. Now there's a thing.

Topic: Networking

Manek Dubash

About Manek Dubash

Editor, journalist, analyst, presenter and blogger.


As well as blogging and writing news & features here on ZDNet, I work as a cloud analyst with STL Partners, and write for a number of other news and feature sites.


I also provide research and analysis services, video and audio production, white papers, event photography, voiceovers, event moderation, you name it...


Back story
An IT journalist for 25+ years, I worked for Ziff-Davis UK for almost 10 years on PC Magazine, reaching editor-in-chief. Before that, I worked for a number of other business & technology publications and was published in national and international titles.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

5 comments
Log in or register to join the discussion
  • Great article Manek. A couple of things though... true object storage is not a file system but a replacement that is massively scalable and gets rid of the inherent problems with file systems... Not sure about the last part of your article referring to acquisitions since Dell is apparently using the Caringo product for their DX system. http://www.theregister.co.uk/2010/03/26/dell_caringo/
    larrysmith123
  • Samy8 is spot on. The Dell Object Storage solution, the DX6000 platform, is Caringo software running on Dell hardware. You can even see it if you look at the console in some of the videos and material. It says "powered by Caringo".
    perri-2e400
  • Agreed except that Gluster doesn't have any object storage back-end, they probably need one to be considered as a highly scalable solution. Also you mentioned object file system offering for Dell but if it's object, it's not file, this a complete different approach, different from block and file as you pointed at the beginning of your article with traditional file system limitations.
    pnico05@...
  • I didn't know at the time of writing that the Dell system used Caringo software - thanks for pointing that out.
    Manek Dubash
  • Good read on object storage, Manek.
    Let me know if you'd like to talk to Amplidata regarding object storage.
    cli-7e479