The end of the RAID era

The end of the RAID era

Summary: The post-RAID era has begun. While RAID arrays aren't going away, the growth is elsewhere, and corporate investment follows growth.

TOPICS: Storage

Why now?

Architecturally superior alternatives to RAID now exist. The post-RAID milestone was passed years ago.

  • The authors of the 1988 original RAID paper (Patterson, Gibson and Katz) all moved on long ago: Patterson to scale-out object storage and much more; Gibson to Panasas, a scale-out object storage company he co-founded; and Katz has been working on Hadoop among many other projects.
  • What are probably the fastest growing large storage infrastructures in the world - Google's and Amazon's - are aren't based on RAID.
  • Major storage vendors including NetApp, HP, EMC and Hitachi have all invested in - and are selling - noRAID systems.
  • But the biggest reason? The math behind erasure codes improved after the RAID paper was written.

How erasure codes work

RAID uses erasure coding to create parity information that protects a RAID array from 1 (RAID5) or 2 (RAID6) uncorrectable read errors (URE). But RAID 5 stopped working 3 years ago if you use SATA drives.

Erasure codes break up a data segment into n fragments, add m additional fragments, store n+m across different devices, and can recover the data from any n of the devices. In a RAID5 8 drive stripe, the original data is divided into 7 fragments, an 8th fragment is calculated - the parity data - and then any one of the 8 drives can fail without losing (theoretically) any data.

The RAID5 problem is that with larger disks rebuild times get dangerously long and that an URE will be found on another disk, killing the rebuild. Surviving 2 failures is the minimum today.

In the '90s a new form of erasure coding was developed that enabled developers to create codes with an arbitrary level of redundancy - survive 4 failures? 10? Pick a number! - called fountain or rateless erasure codes. Startups including Digital Fountain, Cleversafe and Amplidata sprang up to take advantage of these new codes.

This StorageMojo video explores the advantages of rateless codes, using Amplidata as an example. One key advantage: the redundancy needed to survive 4 failures is, they tell me, down to 50-60% of the data. Much better than the 3x replication that Amazon and Google use in their infrastructures, and very competitive with RAID6.

The Storage Bits take Redundant Arrays of Inexpensive Disks shook up a complacent industry 2 decades ago. But time and technology move on.

Make no mistake: RAID isn't going to disappear. But it is moving to the margins as its limitations and costs become clearer thanks to new alternatives.

Despite the industry investment in RAID, we now have better solutions. Properly priced and marketed, these solutions will drive the next big round of storage growth.

Courteous comments welcome, of course. I've been doing work for Amplidata. And for a quick and deeper intro to erasure coding for storage , check out Prof. Jim Plank's Erasure Codes for Storage Applications (pdf) presentation.

Topic: Storage

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • What about the little guy?

    Raid may be passed it time for big data solution shops - but what about home and personal computers? Is there a better solution than RAID currently available or are we assuming SSD performance will make RAID obsolete?
    • RAID ain't about performance

      RAID is about redundancy so your data is safer. What was often referred to as RAID0 was not a RAID setup but a stripe setup where two drives works as one (basically the RAID5 setup with the parity drive missing).

      For home use I would look into new things like Storage Spaces that will ship with Windows 8 and its Windows Server sibling. Storage space can be configured in different redundancy settings (two copies, parity) and does not require disks of the same size as RAID did and can easily be upgraded by simply adding disks when space runs out (even USB ones).

      I'm looking for this for my personal server at home.
      • I agree with lepoete73

        I have used RAID for years for both home and business use, but ONLY for the purpose of redundancy. But now the new next gen file systems sprouting up everywhere are going to offer both performance and redundancy without all of the overhead of RAID AND without the huge Achilles heel that Robin has long pointed out regarding use with high capacity drives. Because of that problem, I have never used drives over 500GB, and even with those drives a RAID restore takes hours. With the next gen file systems, restores from drive errors will be all but instantaneous with very little or more likely no risk of catastrophic failure in the process. After long being a RAID advocate, I now agree with Robin. RAID is going to die from slow attrition as users gradually migrate to newer and more robust solutions.
        George Mitchell
      • Thx!

        Thanks for the info - much obliged!
      • Re: "redundancy so your data is safer"

        No it is not! Keeping your data safe requires BACKUP redundancy, which RAID does not offer! The only kind of redundancy that RAID is good for is the kind that minimizes downtime--if a disk crashes, you can replace it and continue using the system while the array rebuilds, without having to wait to restore stuff from a backup. But it doesn't IN ANY WAY diminish the need for backups!
        • ldo17 .. back that truck up

          I'm pretty sure the point being discussed was home users (esp. average Jane & Joe). How many non-tech folk do you know that manage complex RAID & backup facilities.

          I mean, really? RAID is fine for most folk at home - so long as they have some techie friend or professional who can configure and manage it for them. I'd advise most to go with RAID 1, for simplicities sake.

          Harris is, i'm quite sure, discussing repercussions for business / enterprise. The case for domestic users is more a discussion point for the replies section. But that's obviously not to preclude anyone involved with business / enterprise going into depth about how this discussion relates to the "big boys" (i.e. govt, enterprise, corporations, industrial implementations).
    • Re: What about the little guy?

      The "little guy" gets to use next-generation filesystems, like btrfs for Linux or HAMMER for *BSD. Or Sun's ZFS, if you can find a mass-market OS that supports it.

      All these proprietary big-iron products are hig-margin, low-volume, niche stuff. Ignore them. They'll be rendered obsolete by mass-market products within 10 years.
  • Rumours of my death are ...

    ... dependent upon:

    1. The new technology being available down to the consumer PC, rather than being hoarded at great profit by the usual greedy corporates.

    2. Enterprise markup on storage technologies like RAID 6 ... coming down from the usual 10x factor.

    3. APPL releasing a ZFS file system. MSFT advancing at something less than a snail's pace from the pits of Widnows Home Server (failed file system pseudo-cutdown ZFS), thru' Storage Spaces, to ... something excellent in function and value. (Value to the CONSUMER ... just in case you got confused with SHAREHOLDER there.)

    4. Adrian Kingsley Hughes and Robin Harris putting SSD's in their main working systems ;-)
    ZDNET cloud advocates trying to persuade me that AMZN storage is value for money. Please do not insult my intelligence!
    Ouch. Sorry. Not.
  • Is this implemented in a new file system or is it some custom software?

    How are these systems being implemented? Is there a product out there for enterprise use?

    I searched for noRAID and all I found was "Northern Aid Committee is an Irish American." I don't think that's the same thing...

    Can a small business owner benefit from this information or is it too early?
    • Small Business Implementation of Rateless Erasure Coding technology

      I agree with T1Oracle. How can Small to Medium sized businesses take advantage of this new technology or is this still in the realm of the Enterprise business for now? And Johnfenjackson@ also made great sense. Points 1, 2 and especially 3 are very necessary for this new technology to supplant RAID. Say what you want, but small business and even consumers can understand RAID technology. This new kid needs to make himself easily understood and even more easily available to be the new king of online hard disk data protection. We'll see...
      • Small business and scale-out storage

        Right now rateless erasure codes are being applied to large - hundreds of gigabytes - storage systems. But I expect that to change as new companies tie these codes to very low cost compute & storage nodes.

        One company I like for small to medium businesses - and who is a former client - is Nimble Storage. Their box cost-effectively combines flash and disk, plus they have a 21st century support model that helps overworked IT folks keep up with what the storage is doing. Drobo also has cost-effective and easy to manage file servers.

        BTW, I don't agree that most consumers or even most small businesses understand RAID. They've been told it will protect their data, so it sounds good, but when there is a disk failure they don't understand the rebuild overhead and end up unhappy.
        R Harris
        • "large - hundreds of gigabytes"

          I've got news for you: consumers can aleady buy retail hard drives with THOUSANDS of gigabytes in capacity. If you think "hundreds" of GB is "large", then maybe you're still working with IBM mainframes or something.
  • Becomes more important as drive quality declines

    Our rate of unit failures of so-called "enterprise" drives has tripled over the last couple of years. And it doesn't seem to matter what brand we buy, all of the drive makers have cut so many corners that quality and reliability are at the bottom of the list.

    So RAID and mirroring are essential just to keep our systems running. Moving to new technology has now become a top priority because the failure trend just keeps going up, and the drive makers don't really seem to have any answers.
    terry flores