Why RAID 6 stops working in 2019
Summary: Three years ago I warned that RAID 5 would stop working in 2009. Sure enough, no enterprise storage vendor now recommends RAID 5. Now it's RAID 6, which protects against 2 drive failures. But in 2019 even RAID 6 won't protect your data. Here's why.
Three years ago I warned that RAID 5 would stop working in 2009. Sure enough, no enterprise storage vendor now recommends RAID 5.
They now recommend RAID 6, which protects against two drive failures. But in 2019 even RAID 6 won't protect your data. Here's why.
The power of power functions I said that even RAID 6 would have a limited lifetime.
. . . RAID 6 in a few years will give you no more protection than RAID 5 does today. This isn’t RAID 6’s fault. Instead it is due to the increasing capacity of disks and their steady URE rate.
Late last year Sun engineer, DTrace co-inventor, flash architect and ZFS developer Adam Leventhal, did the heavy lifting to analyze the expected life of RAID 6 as a viable data protection strategy. He lays it out in the Association of Computing Machinery's Queue magazine, in the article Triple-Parity RAID and Beyond, which I draw from for much of this post.
The good news: Mr. Leventhal found that RAID 6 protection levels will be as good as RAID 5 was until 2019.
The bad news: Mr. Leventhal assumed that drives are more reliable than they really are. The lead time may be shorter unless drive vendors get their game on. More good news: one of them already has - and I'll tell you who that is.
The crux of the problem RAID arrays are groups of disks with special logic in the controller that stores the data with extra bits so the loss of 1 or 2 disks won't destroy the information (I'm speaking of RAID levels 5 and 6, not 0, 1 or 10). The extra bits - parity - enable the lost data to be reconstructed by reading all the data off the remaining disks and writing to a replacement disk.
The problem with RAID 5 is that disk drives have read errors. SATA drives are commonly specified with an unrecoverable read error rate (URE) of 10^14. Which means that once every 200,000,000 sectors, the disk will not be able to read a sector.
2 hundred million sectors is about 12 terabytes. When a drive fails in a 7 drive, 2 TB SATA disk RAID 5, you’ll have 6 remaining 2 TB drives. As the RAID controller is reconstructing the data it is very likely it will see an URE. At that point the RAID reconstruction stops.
Here's the math: (1 - 1 /(2.4 x 10^10)) ^ (2.3 x 10^10) = 0.3835
You have a 62% chance of data loss due to an uncorrectable read error on a 7 drive RAID with one failed disk, assuming a 10^14 read error rate and ~23 billion sectors in 12 TB. Feeling lucky?
RAID 6 RAID 6 tackles this problem by creating enough parity data to handle 2 failures. You can lose a disk and have a URE and still reconstruct your data.
Some complain about the increased overhead of 2 parity disks. But doubling the size of RAID 5 stripe gives you dual disk protection with the same capacity. Instead of a 7 drive RAID 5 stripe with 1 parity disk, build a 14 drive stripe with 2 parity disks: no more capacity for parity and protection against 2 failures.
Digital nirvana, eh? Not so fast, my friend.
Grit in the gears Mr. Leventhal points out is that a confluence of factors are leading to a time when even dual parity will not suffice to protect enterprise data.
Consider:
- Long rebuild times. As disk capacity grows, so do rebuild times. 7200 RPM full drive writes average about 115 MB/sec - they slow down as they fill up - which means about 5 hours minimum to rebuild a failed drive. But most arrays can't afford the overhead of a top speed rebuild, so rebuild times are usually 2-5x that.
- More latent errors. Enterprise arrays employ background disk-scrubbing to find and correct disk errors before they bite. But as disk capapcities increase scrubbing takes longer. In a large array a disk might go for months between scrubs, meaning more errors on rebuild.
- Disk failure correlation. RAID proponents assumed that disk failures are independent events, but long experience has shown this is not the case: 1 drive failure means another is much more likely.
Simplifying: bigger drives = longer rebuilds + more latent errors -> greater chance of RAID 6 failure.
Mr. Leventhal graphs the outcome:
Courtesy of the ACM
The Storage Bits take For enterprise users this conclusion is a Big Deal. While triple parity will solve the protection problem, there are significant trade-offs.
21 drive stripes? Week long rebuilds that mean arrays are always operating in a degraded rebuild mode? Wholesale move to 2.5" drives? Functional obsolescence of billions of dollars worth of current arrays?
Home users can relax though. Home RAID is a bad idea: you are much better off with frequent disk-to-disk backups and an online backup like CrashPlan or Backblaze.
What is scarier is that Mr. Leventhal assumes disk drive error rates of 1 in 10^16. That is true of the small, fast and costly enterprise drives, but most SATA drives are 2 orders of magnitude less: 1 in 10^14.
With one exception: Western Digital's Caviar Green, model WD20EADS, is spec'd at 10^15, unlike Seagate's 2 TB ST32000542AS or Hitachi's Deskstar 7K2000 (pdf).
Comments welcome, of course. Oddly enough I haven't done any work for WD, Seagate or Hitachi, although WD's indefatigable Heather Skinner is a pleasure to work with. I did work at Sun years ago and admire what they've been doing with ZFS, flash, DTrace and more.
Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.
Talkback
DIY
Work in progress is on RAID 1 and I have an AKASA Duodock into which I can plug naked SATA disks. When the RAID array reports 'degraded' I simply copy the remaining operational drive to a cold spare. 'Tested' this out for real over the two hard disk failures I've had in the last 6 months.
However, many OEM PC's only support two drives so I'm thinking of repurposing two old machines in larger cases (6/8 drives) for virtualised storage. Interested to hear your comments on:
http://talkback.zdnet.com/5208-12695-0.html?forumID=1&threadID=75503&messageID=1468605&tag=content;col1
from your colleague Kusnetzky's thread.
That way bulk storage can be secured whilst (some) WIP can be on faster SSD's.
RAID with regular backups is the most secure option ...
But having said that, ZFS type filesystems offer the ultimate solution in that they can detect and correct problems quickly without needing long rebuild times. However, there again, if your drives aren't being read on a regular basis as in regular complete backups, even ZFS type systems will risk sleeper problems where one or more clusters deteriorate and then the other drive fails leaving you sunk. I see a high level of risk with "lazy" disk drives. As with the human body, a little exercise pays huge dividends in the long run.
RE: Why RAID 6 stops working in 2019
RE: Why RAID 6 stops working in 2019
Never did feel that RAID was necessary for home use.
RE: Why RAID 6 stops working in 2019
RE: Why RAID 6 stops working in 2019
home users. Why? Because they understand files and file
interfaces are simpler. Block devices don't compute for
many people and iSCSI drivers + setup are usually non-
trivial.
Whether you go block or file though the key to personal
data preservation is to maintain multiple copies of your
data. Any single device can and will fail. You can't let it
take your only copy when it does.
Robin
Drobo?
I bought a Drobo S and am trying it out.
considering how well it works for civilians.
Expect a report in the next couple of months.
Robin
My comments on using Drobo
So the question is, how will you use the Drobo?
There are a few ways:
1. Direct-connect to PC via USB or eSATA.
2. Network-connected via ethernet with DroboShare device.
3. iSCSI over network. (I don't think all Drobo units have this.)
So for the home user it comes down to direct-connect (one PC/Mac) or ethernet (all PCs/Macs on network).
The Drobo itself is wonderfully simple to use, and is very well-constructed. It has a nice magnetic cover for accessing the drives and big bright lights to clearly point out good and bad status.
It is simple to add or delete drives. Just plug 'em in.
The Drobo software for managing the device is OK, but not as good as the device it is managing. My Drobo has the DroboShare device, and it requires me to keep tabs on three different types of updates: the Drobo firmware, the DroboShare firmware, and the Drobo management software. When I first plugged everything in, it required me to check for updates a number of times before everything was up to date.
You have a bit of a learning curve with regard to keeping an eye on disc usage. When you setup the Drobo, you give it a target size for your array, not the actual amount of storage you have today. So you might only have 2TB in actual disc storage, but you can create an 8TB array. It's mostly a good thing, once you get the hang of it.
The biggest drawback I have seen, as someone who attaches the Drobo via ethernet (DroboShare) is that it is not the speediest storage in the world. Sometimes it can be very slow. (NOT due to running out of storage space. It has a feature that automatically slows down writes as it gets close to filling up, but I am only about one-quarter full in terms of actual on-hand disc space.)
Many times when I try accessing the Drobo there is an initial pause, and then it reacts with typical performance. It might have something to do with the way it monitors the network, I'm not sure. But the pause is a little too long for my likes.
My "rough estimate" of data transfer speed is that it takes maybe twice as long to copy a bunch of files to the Drobo than to copy the same files to a Windows Server file share. (I'm sure Robin can do much more accurate testing than my "gut feel" described here.)
On the plus side, the plug-and-play aspect to network storage was great with the Drobo. It worked the first time, with absolutely no tweaking on my part. (Aside from learning how to use the sometimes-confusing Drobo management software.)
The DroboShare unit has a bit of a bonus for techies: You can use it to run small applications. There are a handful of apps on the Drobo site, and some are mildly useful. I played around with the feature for about a day, and came to the conclusion that it is a cool idea, but because it would require me to learn a new technology ecosystem, and the scope of such is limited to just Drobo users, it makes more sense to build the apps to sit on some other platform.
That said, it does provide an opportunity for industrious people to build apps that can turn the Drobo into a functional stand-alone server that can handle more than mere file sharing.
I like the Drobo for its ease-of use, peace-of-mind, and at-a-glance view of its status. It's great for storing shared files, bulk storage, backups, and other things that are not ultra performance-sensitive.
However, for those looking for a "data drive" on their PC that will be accessed frequently, I'm not so sure. Since my Drobo is set up as network storage I have not tested extensively in a direct-connect scenario. My guess is that there are faster direct-storage units out there, but Robin could test that in his upcoming review.
Bozoshare?
And DataCenter storage designers go big
RAID's original meaning is still true. It would be better to have 15 SANs of 2 TB each than a single 30 TB one even with the cost of extra hardware - you now have no single point of failure and alternate plans and methods of recovery if one goes down. 5000 people unable to work for one week is $5 Million - vs cost of extra hardware - which really will cost more?
You're forgetting something about SANs
Additionally SAN designers are NOT USING SATA except for temporary storage or dedupe backups. They are typically using 300/600 GB Fiber Channel disks in the big SANS. The spindle speed is typically 10,000 or 15,000 RPM for fast rebuilds, fast scrubbing, and fast access.
I assume, Robin, that you are targeting consumer RAID and very small business RAID. Even small business is typically purchasing 10K and 15K enterprise class SAS disks that are on the order of 146 - 300 GB with UREs every 10^15.
The situation is very different in the enterprise where people are typically not storing huge iTunes and movie libraries.
Learned somethign new -- again!
ZFS --> Mainstream in future
Licensing issues? I was hoping Apple would license and move forward
with ZFS. Now its not on the radar.
Perhaps things will change and ZFS will again gain some support for
future use in mainstream OS file systems.
What about MTBF
Yes, the MTBF but a different point about it
I guess this could be circumvented by not using one single batch of one single kind of disk of one specific manufacturer.
I've seen a bulk of disk fail around the same time.
But to come back to the point you are making. 2019 is a bit too optimistic, i concur ;)
You are ignoring sympathetic failures then
Good point
I use MTBF to compare...
MTBFs mean very little...