Maybe this is an opportunity for software designers to create and implement data integrity applications similar to the rise of security applications that are all the rage today.
By and large, most software today does not have any data integrity logic built into it. Data is assumed to be correctly handled by the underlying I/O routines and the hardware. But we all know that there are data integrity problems with both the operating systems and the hardware, it's just that nobody ever does anything about it.
Part of the reason is training and education; current computer science training doesn't spend much time on data integrity. When you talk to people trained in the '60s and '70s, this was a much higher priority. Storage mediums like cards, tape, and early disk systems were physically prone to errors (tape stretching, card jamming, disk head misalignment). Programmers learned and used techniques like parity and CRC checking, redundant data groups, and other data recovery algorithms. Today, everybody expects the OS or the hardware to do it all automatically.
A big factor in the failure is the loose standard of the PC architecture. It's impossible for vendors to do rigorous integration testing of components and subsystems in today's markets, there are so many combinations and variants, and testing adds costs that people aren't willing to pay. The same is true for disk vendors. They are under incredible price pressures even as they roll out new technical innovations every few years. Size matters, quality is a distant second concern.
Finally, Microsoft bears some of the blame for this problem, because they refuse to acknowledge it. I did a simple demonstration for our CIO awhile back, when he was deciding if we would use Windows or Unix/Linux for our server architecture. I set up a Windows server and a Unix server and ran our software to generate a load on the systems. Then I took a data CD, scratched it with a car key, and put it in the drive on the Windows server. Windows promptly locked up for several minutes while it tried to read the scratched disk, freezing all access to the software application. Then I put the scratched CD in the Unix system, and after a few seconds, a window popped up on the console informing us of an error reading the CD, while the system kept humming along. It took the CIO about 30 seconds to make the platform decision ...
Many people reacted with disbelief to my recent series on data corruption (see How data gets lost, 50 ways to lose your data and How Microsoft puts your data at risk), claiming it had never happened to them. Really? Never had to reinstall an application, an OS, or had a file that wouldn’t open?
Are you sure?
The research on silent data corruption has been theoretical or anecdotal, not statistical. But now, finally, some statistics are in. And the numbers are worse than I’d imagined.
Petabytes of on-disk data analyzed
At CERN, the world’s largest particle physics lab, several researchers have analyzed the creation and propagation of silent data corruption. CERN’s huge collider - built beneath Switzerland and France - will generate 15 thousand terabytes of data next year.
The experiments at CERN - high energy “shots” that create many terabytes of data in a few seconds - then require months of careful statistical analysis to find traces of rare and short-lived particles. Errors in the data could invalidate the results, so CERN scientists and engineers did a systematic analysis to find silent data corruption events.
Statistics work best with large sample sizes. As you’ll see CERN has very large sample sizes.
The program
The analysis looked at data corruption at 3 levels:
- Disk errors.The wrote a special 2 GB file to more than 3,000 nodes every 2 hours and read it back checking for errors for 5 weeks. They found 500 errors on 100 nodes.
- Single bit errors. 10% of disk errors.
- Sector (512 bytes) sized errors. 10% of disk errors.
- 64 KB regions. 80% of disk errors. This one turned out to be a bug in WD disk firmware interacting with 3Ware controller cards which CERN fixed by updating the firmware in 3,000 drives.
- RAID errors. They ran the verify command on 492 RAID systems each week for 4 weeks. The RAID controllers were spec’d at a Bit Error Rate of 10^14 read/written. The good news is that the observed BER was only about a 3rd of the spec’d rate. The bad news is that in reading/writing 2.4 petabytes of data there were some 300 errors.
- Memory errors. Good news: only 3 double-bit errors in 3 months on 1300 nodes. Bad news: according to the spec there shouldn’t have been any. Only double bit errors can’t be corrected.
All of these errors will corrupt user data. When they checked 8.7 TB of user data for corruption - 33,700 files - they found 22 corrupted files, or 1 in every 1500 files.
The bottom line
CERN found an overall byte error rate of 3 * 10^7, a rate considerably higher than numbers like 10^14 or 10^12 spec’d for components would suggest. This isn’t sinister.
It’s the BER of each link in the chain from CPU to disk and back again plus the fact that for some traffic, such as transferring a byte from the network to a disk, requires 6 memory r/w operations. That really pumps up the data volume and with it the likelihood of encountering an error.
The Storage Bits take
My system has 1 TB of data on it, so if the CERN numbers hold true for me I have 3 corrupt files. Not a big deal for most people today. But if the industry doesn’t fix it the silent data corruption problem will get worse. In “Rules of thumb in data engineering” the late Jim Gray posited that everything on disk today will be in main memory in 10 years.
If that empirical relationship holds, my PC in 2017 will have a 1 TB main memory and a 200 TB disk store. And about 500 corrupt files. At that point everyone will see data corruption and the vendors will have to do something.
So why not start fixing the problem now?
Comments welcome, of course. Here’s a link to the CERN Data Integrity paper. CERN runs Linux clusters, but based on the research Windows and Mac wouldn’t be much different.
Doesn't mean I don't understand the history of the industry or how things are now.
