WWDC's biggest disappointment

WWDC's biggest disappointment

Summary: There were many positive developments unveiled at Apple's recent WWDC. But one major piece of 1980s technology at the center of the Mac and iOS stack needs replacement. Tech rarely gets better with age.

SHARE:

Can you think of another technology from 1985 that is still central to your Mac? Me neither.

But HFS, the Mac/iOS file system, dates from 1985 - the early years of the Mac - and was lightly updated to become HFS+ in 1998, three years before Mac OS X debuted. HFS+ gave us 32 bit block addresses, longer file names and Unicode, but the underlying architecture remains 1985.

That's why the biggest disappointment of this year's WWDC is that no new file system was announced. Apple knows they have a problem: they announced ZFS on Mac Server back in 2007 before licensing issues and a lawsuit caused them to decommit.

Right. Seven years ago. And still nothing better.

What's the problem?
The problem is that with users commonly storing millions of files, the bit rot inherent in storage - remember, the Universe hates your data - goes uncorrected and undetected. Until you try to access the file and you can't.

At this point someone will comment that they've never seen the problem. Of course, how could you unless you try to open every file on your system? Or realize that when an app or file won't open it is because of bit rot.

That's why I find this recent post from Aymeric Barthe especially interesting. He compared 15,264 photos that he'd kept for years on both his Mac and online. He wrote a script to compare the two sets of files.

I ran the shasum commmand line tool to compute SHA1 hashes of every single file in the backup folder, except .DS_Store files. Then, I ran shasum in verify mode to check the files on my main volume against the hashes. Differences either indicate voluntary modifications (which did not apply in my case), or corruptions courtesy of HFS+ (which was my case).

He found that HFS+ had corrupted 28 files. The right answer for a file system is zero. Not a large number, you think? What if one of them was your favorite family photo or video?

The Storage Bits take
You can test HFS+ yourself. It simply isn't up to the standards of modern file systems such as the superior technology of Microsoft's NTFS.

That's why I'm disappointed that Apple hasn't replaced HFS+. Unix was designed so new file systems could be easily dropped in.

But on the Mac we'll be celebrating(?) HFS's 30th anniversary in a few short months. It's time for a change.

Comments welcome, of course. 

Topics: Storage, Apple, Microsoft, Software

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

11 comments
Log in or register to join the discussion
  • NTSF is no better on BIT rot.

    I have had similar issues with both. ZSF being in limbo is the worst aspect of Oracle's purchase of Sun.
    Bruizer
    • NTSF is no better on BIT rot.

      Yes, I would welcome ZFS provided it also provides, particularly for block de-duplication. But I wouldn't lose Fusion to get it. And yes, I know that ZFS can use SSD's as well. It just doesn't use then as intelligently.

      "...and was lightly updated to become HFS+ in 1998, three years before Mac OS X debuted"

      But then he forgets (or deliberately omits) to mention that Journaling and case sensitivity were added in November 2002, post Mac OS X. Journaling provided a major improvement on data reliability so clearly should have been mentioned in context.

      The ACLs were added at MacOS 10.4 giving far finer control granularity.

      10.5 added UFS's hard links, 10.6 added compression and 10.7 added logical volume encryption.

      And then there is core storage and Fusion which provides HFS aware dual media speed optimisation.

      To represent MacOS' file system as having been static since 1998 is absurd.


      The statement

      "...simply isn't up to the standards of modern file systems such as the superior technology of Microsoft's NTFS."

      ... is garbage

      NTFS was an improvement on FAT. But FAT wasn't even close to state of the art, when it was introduced in 1977. So being an improvement on it wasn't much of a hurdle.

      I've had far more corrupted NTFS file systems than HFS.

      Whatever happened to ephemeral garbage collection?
      Henry 3 Dogg
  • So this really shows that you should have multiple backups

    and you should keep some older backups and sometimes copy the backups to another drive. So how is that news? ...

    My home system (a main computer with two separate boot drives, one running Win 8.1 Pro, the other Win 7 Home Premium, a "spare" running Win 8.1 Pro, and a laptop I rarely use) has about 5.7 TB of storage. That includes a 2TB external backup, a 1.5 external backup, and a 1TB drive partitioned as 200GB that I use and 800GB that is "spare" space not normally used, and a 500GB data drive on the "spare" computer that is almost empty. So I have LOTS of backup space and lots of backups.
    Rick_R
    • Well....

      The problem is the backup has to occur before the bit rot and then not overwritten after the bit rot. And, when one recovers the corrupted file, they have to find the archive that preceded the corruption.

      But, kids, don't not back up. Then you've added a huge problem to the one minor and somewhat random one.
      DannyO_0x98
  • Pretty insignificant

    If this is the biggest "disappointment", there truly is extremely little to complain about. And naturally, the blogosphere must find *something* to complain about.
    monoclast@...
  • Less than 2 files per thousand...

    ...after "years" doesn't sound so bad to me. Zero sounds better, but unrealistic, I think.
    pdq
  • About NTFS

    The author claimed that, "[HFS+] simply isn't up to the standards of modern file systems such as the superior technology of Microsoft's NTFS."

    I don't believe that's true. NTFS predates HFS+ by 5 years. HFS+ supports all of the features and functionality that NTFS does. Wikipedia has a good comparison of file systems article.
    deasys
  • HFS isn't great, but not as bad as you think...

    I'm not going to make the case that I wouldn't want Apple to come out with a new OS with better data integrity capabilities. However, to suggest that Apple has done nothing with HFS since 1985 is just silly.
    Apple has evolved HFS considerably over the years. Much of Apple's design decisions have been in the name of backward compatibility. Further, many new features that are associated with the operating system are being handled through Core Storage.
    Since HFS+ in 1998, Apple has added Journaling (OS X 10.3), case sensitivity, ACL based file security, directory hard linking (10.5), file compression (10.6), logical volume encryption - through core storage (10.7).
    The direction Apple seems to be headed is in favor of core storage which sits on top of and is independent of the file system. As more APIs rely on core storage, swapping out file systems will eventually become trivial.
    techconc
  • I have several images with partial sections magically turned pink or green

    A couple years ago I discovered a few image files have partial sections magically turned pink, green or garbled pixels due to bitrot when viewed in some Windows image viewer. And all of them are files stored on .... NTFS formatted partitions!!!

    How do I know they are bitrot?

    Windows did not report any checksum or sector read error when opening those images so Windows thinks the file are okay. However, the program I used to open those images did report image corruption though.

    But I did not alter those images and all photo collections in my library have read-only attribute turned on *AND* deny write permission set for everyone in NTFS to prevent accidental modification.

    When I realized a few images files have been corrupted, I was able to restore them from a two years old backup from a external hard drive. And guess what, the restored images are okay.

    So bitrot confirmed and NTFS was no help in preventing or correcting those errors.

    Mr. Harris.. If you want to discuss how popular files systems we use do not prevent bitrot, why only discuss Apple's HFS+?

    Why would you even say something like "HFS+ ..... isn't up to the standards of modern file systems such as the superior technology of Microsoft's NTFS" **without** pointing out that the 20+ years old NTFS does not prevent nor correct bitrot either?

    Everyone agree that is a biased opinion by the author?

    As for my terabytes of file collections, I will make do with NTFS plus SnapRAID.
    CaptainGoodnight
  • Not

    I would say ghat much more of my files from last 15 years have been rendered unusable in the Windows world than the mac and much more of an issue than bit rot is non-supports files types. This is more true of Adobe than anyone else.
    Just try and open than pagemaker or illustrator file from the early days of windows or Mac OS9/OSX.
    And most of my files from 3.5.1 Nt are unusable!
    kpbpsw
  • I am wondering if solid state drives

    don't kind of negate the need for this? Although SSD opens a lot of doors for newer file systems (I wonder if the SQL based WinFS might not have survived if it were in development now?), it also makes the problems of older file systems a lot less exacerbated.
    Mac_PC_FenceSitter