File systems guard all the data in your computer, but most are based on 20-30 year old architectures that put your data at risk with every I/O. The open source ZFS from Sun Oracle claims high data integrity - and now that claim has been tested.
I’m at the USENIX File and Storage Technology FAST conference (see the last couple of StorageMojo posts for more) in Silicon Valley. There is more leading edge storage thinking presented here than any other industry event.
Case in point: End-to-end Data Integrity for File Systems: A ZFS Case Study by Yupu Zhang, Abhishek Rajimwale, Andrea C. Arpaci-Dusseau and Remzi H. Arpaci-Dusseau of the Computer Sciences Department, University of Wisconsin-Madison. It offers the first rigorous test of ZFS data integrity.
Methodology
The UW-M team used fault injection to test ZFS. Fault injection is a great technique because you inject as many errors as you want and correlate the file system’s response.
Goal: errors injected = errors rejected.
As the paper puts it:
. . . we analyze a state-of-the-art file system, Sun Microsystem’s ZFS, by performing fault injection tests representative of realistic disk and memory corruptions. We choose ZFS for our analysis because it is a modern and important commercial file system with numerous robustness features, including end-to-end checksums, data replication, and transactional updates; the result, according to the designers, is “provable data integrity”
Disk corruption is more common than you think.
In a recent study of 1.53 million disk drives over 41 months, Bairavasundaram et al. show that more than 400,000 blocks had checksum mis- matches, 8% of which were discovered during RAID reconstruction, creating the possibility of real data loss.
But the study didn’t stop there. They also injected errors into the RAM the file system used. As regular readers know, memory errors are hundreds to thousands of times higher than thought.
Keepin’ it real
ZFS has several important data integrity features.
- Data integrity checksums. The checksums are stored in parent blocks, which enables ZFS to detect silent data corruption.
- Block replication for data recovery. ZFS keeps replicas of certain important blocks.
- Copy-on-write for atomic updates. Object updates are grouped together and new copies are created for all the modified blocks. ZFS doesn’t need journaling.
ZFS has no special tools against RAM errors.
Conclusion
The study found that
. . . ZFS successfully detects all corruptions and recovers from them as long as one correct copy exists. The in-memory caching and periodic flushing of metadata on transaction commits help ZFS recover from serious disk corruptions affecting all copies of metadata.
The results for in-memory data corruption weren’t as stellar, but ZFS - like every other file system - wasn’t designed to handle DRAM errors. The authors offer suggestions for making ZFS less vulnerable to DRAM errors.
The Storage Bits take
The disk tests are the strong evidence that ZFS delivers on its promise of superior on-disk data integrity. I hope that ZFS - or something better - arrives on other OS’s soon.
But what about the memory fault tests? In this I suspect that ZFS is no worse than legacy file systems and as a clean sheet design may be better.
While suggestions for improved RAM error resistance are well taken - after all, most of the world’s computers have no ECC memory - this reinforces the need for ECC DRAM in important applications. ECC everywhere!
The authors should have made the effort to correlate DRAM error rates to the likelihood of in-memory data corruption by file systems. While DRAM is not nearly as reliable as the industry let us believe, we know disks are prone to errors.
The companies who produce file systems are fortunate their failures do not end up - like Toyota - with a grandmother wrapped around a tree. But what they lack in drama they make up with volume.
We have no idea how many billions of man-hours have been wasted due to silent data corruption, but the number will keep growing until every file system is at least as good as ZFS is today.
That means you, NTFS, HFS+ and the other legacy Unix, Linux and proprietary file systems. We trust you with our data and you are letting us down. That’s just wrong.
Kudos to the UW team for their testing and their paper. They’ve set the bar for the rest of the industry.
Comments welcome, of course. See ZFS: Threat or menace? for a detailed introduction to ZFS. For the impact of data corruption see How Microsoft puts you data at risk. The NTFS team knows this stuff, but the MSuits are more worried about the Zune.
