Apple's new kick-butt file system

As a long time fan of Apple - I bought an Apple // in 1978 - I watch Apple's storage efforts with special interest. The least talked about addition to the next version of Mac OS X, Leopard, is notable. Especially since Microsoft's WinFS bit the dust.

Apple is doing something really cool with storage - not to discount their laudable RAID product - and that something is called ZFS. The bright side of the Leopard slip: more time to integrate ZFS is a Good Thing.

ZFS = non-acronym ZFS is a very cool - and open source - file system that some smart guys at Sun built. Its tree structured checksums eliminates most of the bit rot that afflicts Macs and PCs. When ZFS retrieves your data, you can be sure it is your data, and not the misbegotten spawn of a driver burp.

Add a disk drive to ZFS and it simply joins the pool of blocks available for storage. You don't have to manage another disk.

It is cheaper for ZFS to do a snapshot copy than it is to overwrite your data. While Time Machine doesn't require ZFS - journaling HFS+ can do it, ZFS would make it easier and perform better.

Here's some more cool stuff

Here are the highlights of some of the changes you'd see with ZFS on Leopard, the next version of the Mac OS.

No more Disk Warrior Data corruption on PCs and Macs is a sad and stupid fact of life. Power failures, flaky RAM, poor grounding, (slowly) failing hard drives, driver glitches, phantom writes and more conspire to rot your data.

ZFS eliminates that. All blocks are checksummed and the checksum is stored in a parent block. ZFS always knows if the block is correct and/or corrupt. Every block has a parent block (with one obvious exception that gets special treatment), so the entire data store is self-validating. You'll never have to wonder if all your data is correct again. It is.

No RAID cards or controllers ZFS implements very fast RAID that fixes the performance knock-off against software RAID. In ZFS all writes are the fastest kind: full stripe writes. And the RAID is running on the fastest processor in your system (your Mac), rather than some 3-5 year old microcontroller.

Just add drives to your system and you have a fast RAID system. With Serial Attach SCSI and SATA drives you'll pay for the drives (cheap and getting cheaper), cables and enclosures.

No more volumes Every time you add a disk to your Mac you see another disk icon on the desktop. If you want to RAID some disks you use Disk Utility (or something) to create the volume. Slow, error-prone, confusing.

ZFS eliminates the whole volume concept. Add a disk or five to your system and it joins your storage pool. More capacity. Not more management.

Backup made easy ZFS does something called snapshot copy, which creates a copy of all your data at whatever point in time you want. Copy the snapshot up to a disk, tape or NAS box and you are backed up.

Create a snapshot on every write if you want, so if your database barfs you can go back to just before it choked.

But that's not all! For in-depth treatment of ZFS see here and here. Includes links to more technical info and benchmarks.

Why does Apple care? After all, journaled HFS+ isn't perfect, but it is competitive with NTFS and the other common filesystems out there. My original thought was "here is this great free product so why wouldn't you use it."

Well, as others have noted, while plugging in a new file system isn't that hard, it does take investment, such as migration, and creating the front ends for all the cool things you can do with ZFS. Steve may not care much about plumbing but he is all over user experience. Migration in particular is difficult for home users who don't have empty external hard drives.

Now we know The motive is clear: HDTV content to feed Apple TV. How does this impact storage?

Video downloads: big and getting bigger! Here's how. Imagine you've built the world's largest and most successful online music store and sold billions of dollars of hardware to play that music. Each of those tracks cost $0.99 and is 3-5 MB each. People can easily back them up and even if they have a few hundred, it is maybe a GB or two. Easy to back up on a few CDs or DVDs. And they are on your iPod anyway. So HFS+ burps on your music and other than yelling at an underpaid Apple tech support guy, what are you going to do? If it wasn't backed up, whose fault is that?

Enter the terabyte media collection Now you want to build the world's largest and most successful online video store, with DVD and HDTV quality content. You are a little ahead of the market, but that usually works out. You want people to buy movies as freely as they now do tracks. Yet there is the scale problem: movie files are 1000x the size of audio or photo files. Not only that, the studios don't want you to back them up to DVD or anything else.

"Halfway through T3 the hard drive started clicking?" iTunes music is automatically backed up if you have an iPod. Movies aren't. Movies are large - 1 to 2 GB today - and much larger with HDTV and DTS sound. If you want people to store and play movies digitally, both purchased and home video, they need safety and capacity. No disk tools. No RAID set-up. No volume management. Suddenly storage quality and ease of use becomes a critical success factor for a new billion dollar business.

ZFS is the answer Steve Jobs has two questions. First, how can I sell more online content and equipment to play it? Second, how can I kick Microsoft's butt? By solving the high-capacity storage problem for HDTV content way better than Microsoft can, he's got a great answer to both questions. He'll never utter "ZFS" to a starstruck MacWorld audience. But he will wheel out a half dozen features, like Time Machine, based on ZFS, that will instantly become must-haves for the home digital media center.

Apple Computer had the means, ZFS; motive, a big market; and opportunity to murder the Media PC.

I expect they'll introduce the way they did HFS+: on OS X server. After they're confident, it will be the default file system. And the folks in Redmond will be scratching their heads once more..

Comments Welcome, As Always I'm off to NAB and SNW this week, so look for updates on cool new stuff.