Why do you want WinFS?

Why do you want WinFS?

Summary: Every time the conversation turns to Windows 7, the subject of WinFS comes up. This next-generation file storage software was cut from Windows Vista (and eventually killed completely) nearly four years ago. But it keeps reappearing on lists of features that people want to see in Windows 7. Why? WinFS made for great PowerPoint slides, but it was a terrible idea. Don't just take my word for it: Some of Microsoft's most experienced and respected developers agree.

SHARE:

One point I made in my wish list for Windows 7 deserves repeating:

Several commenters on [Sinofsky’s] initial “Welcome” post expressed hope that the WinFS file system, which was killed off during the infamous “Longhorn reset,” would be resurrected for Windows 7. Not gonna happen. Nor, frankly, is it necessary.

Last time I looked, at least four commenters on Sinofsky's post had pleaded for the return of WinFS. Within a few hours of my post, my ZDNet colleague Adrian Kingsley-Hughes banged out his Windows 7 wishes. Right smack in the middle of the list was “WinFS …” No further details, just the bare item. In the Talkback section of Adrian’s post, I was intrigued to see this comment from Larry Osterman:

Why do you want WinFS?

What [capabilities] did WinFS give you that Vista doesn't already provide?

Good questions. And the guy doing the asking isn't just some random passer-by, either. Larry Osterman is a veteran Microsoft developer (started in 1984) who has written code for every member of the Windows NT family and who currently works on the core Windows audio engine. (His blog is a must-read if you’re even slightly interested in the Windows Vista audio stack.) Every time I hear someone pining for the return of WinFS, I ask the same question as Larry. Why do you want WinFS? What problem are you trying to solve? Although it made for great PowerPoint slides, WinFS was a terrible idea, and killing it was one of the smartest things Microsoft ever did.

For those who need a little background, WinFS is short for Windows Future Storage. (Wikipedia has a long, dry, technical overview that’s worth reading if you want more details. NTFS.com has a slightly more accessible explanation, complete with diagrams.) It was one of the “gee golly whiz” features of the Longhorn project, which was the original codename for the operating system that became Windows Vista. WinFS was announced at the 2003 Professional Developers Conference (PDC) and actually shipped in a very early beta of Longhorn, back in 2004. Unfortunately, it didn’t work particularly well, an attribute it had in common with a lot of other stuff in those early betas. A few months after that 2004 beta release, the Windows team reviewed the grandiose plans for Longhorn and scaled it back, a process that later became known as the “Longhorn reset.” WinFS was one of the first things cut and it was later killed as an OS component. (If you want to see the hype cycle for this feature in its full splendor, go read the WinFS Team Blog from start to its ignominious finish in June 2006, just before Vista was locked down.)

WinFS is probably the most misunderstood Windows feature of all time. The biggest misconception is that it was going to replace NTFS, the file system that’s used by every member of the Windows NT family, which includes Windows Vista. (The fact that each name ends in FS makes the confusion almost inevitable.) Although it looks and acts like a file system, it isn't; WinFS was intended as a SQL database that sits on top of the underlying file system (which means you could replace NTFS with a different file system or port WinFS to another operating system and it would still work). The idea was to allow applications to share, compare, and aggregate data from different stores.  I saw it demoed several times, publicly and privately, back in 2003 and 2004. All the capabilities I was shown then are in Vista today and will be in Windows 7 when it arrives, using XML, tags, and the systemwide index provided by Windows Search.

Way back in 2004, I wrote about this decision in a post entitled “Longhorn wish list.” (Ironic, eh?) My take at the time:

Honestly, I don’t care that much about WinFS and Avalon. They’re cool, but they can wait.

And they can continue to wait. The reason WinFS was cut is that it didn’t actually solve a user need, it didn’t offer any compelling benefit to developers, and it dragged down overall system performance. Back in June 2004, Microsoft’s Dare Obasanjo nailed this point [emphasis added]:

[T]he core scenarios touted for the encouraging the creation of WinFS (i.e search and adding metadata to files) don't really need a solution as complex or as intrusive to the operating system as WinFS. The only justification for something as radical and complex as WinFS is if Windows application developers end up utilizing it to meet their needs. However as an application developer on the Windows platform I primarily worry about three major aspects of WinFS. The first is performance, I definitely think having a query language over an optimized store in the file system is all good but wouldn't use it if the performance wasn't up to snuff. Secondly I worry about security, Longhorn evangelists like talking up what a wonderful world it would be if all my apps could share their data but ignore the fact that in reality this can lead to disasters. Having multiple applications share the same data store where one badly written application can corrupt the entire store is worrisome. This is the fundamental problem with the Windows registry and to a lesser extent the cause of DLL hell in Windows. The third thing I worry about is that the programming model will suck. An easy to use programming model often trumps almost any problem. Developers prefer building distributed applications using XML Web Services in .NET to the alternatives even though in some cases this choice leads to lower performance. The same developers would rather store information in the registry than come up with a robust alternative on their own because the programming model for the registry is fairly straightforward.

It’s worth noting that Dare wrote that when WinFS was alive and still being vigorously defended. That took some guts. A few years later, another Softie, Matt Warren, called WinFS “the biggest fiasco [he] ever bore witness to.” Back in 2003, he was working on an alternative project called ObjectSpaces. Here’s how he remembers WinFS:

The WinFS client API even started out as a complete copy of the ObjectSpaces codebase and had all the same limitations. It just had more political clout as it was being [led] by a figure at a higher-point in the corporate org chart, and so it was positioned as part of a juggernaut that was making a massive internal land grab. We on the outside used to refer to WinFS as the black hole, forever growing, sucking up all available resources, letting nothing escape and in the end producing nothing except possibly a gateway to an alternate reality. Many of our friends and co-workers had already been sucked in, and the weekly reports and horror stories were not for the weak-of-heart.

[…]

There were not too many in the developer division that believed in the mission of WinFS. As a developer tool for the masses, something simple that targeted the lower end was paramount. ObjectSpaces had been it, and now it was gone. There was still some glimmer of possibility that WinFS v2 might eventually get it right and be useful as a general ORM tool. But all hope of that was shot when WinFS was pulled out of Vista and its entire existence was put in doubt.

The next time someone tells you that WinFS is on their wish list for Windows, do me a favor and send them here. I’m still trying to find someone who can give me even one good reason.

Update: After posting this, I ran across an excellent May 2008 interview that Jon Udell did with Quentin Clark, who ran the WinFS project, and who posted much of the content on the WinFS Blog that I cite earlier. The interview is entitled "Where is WinFS now?" and in it, Clark acknowledges that the Longhorn effort was not the epitome of smooth development: "[W]e got tripped up in the Longhorn cycle. We were building too much of the house at once. We had guys working on the roof while we were still pouring concrete for the foundation. At one point we realized we needed to decouple things. And that really did give this team the freedom to go off and take these underlying technologies, which we believe were fundamental to the database, and get them done correctly."

He also confirms my suggestion that the best home for these technologies, at least for now, is not in the file system but rather in the cloud computing environment. The big news is that the follow-up to WinFS is "shipping in SQL Server 2008, as a feature called filestream." So it will be an add-on that apps can take advantage of, but it is not ready to integrate into the file system today:

Anyway, we did a partnership with Microsoft research, and at some point along the arc we solved it fairly well. It's not trivial. This is not something that ends up being a simple solution to this very complex problem. It's actually reasonably sophisticated, but it works, and we built it in as part of the last WinFS beta.

As they realized they were onto something, they started to fork out a componentized version of it that's now finding its way into a bunch of Microsoft products. The official branding is Microsoft Sync Framework. I think they're on target for shipping it in six different products, and for embedding it all over the place.

If you're interested in this stuff, the whole interview is well worth reading.

Topics: Operating Systems, Microsoft, Software, Windows

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

114 comments
Log in or register to join the discussion
  • same question

    I posted exactly the same question to AKH article talkback. It will be interesting to see the answers.
    trenchsol
    • What about licensing Sun ZFS?

      From what I have seen written about Sun's ZFS, it looks like it would be an amazing replacement for NTFS. It is fast. It has provable data integrity. It has unlimited scalability for the future. It appears to be self-repairing (within reason). I don't profess to be a file system expert but from the layman explanations I've seen regarding how it works, it is one of those "Why didn't somebody think of this sooner?" moments.

      I've also heard Apple was so impressed it is licensing it for a future version of OSX. I believe it already works on Linux. Wouldn't it be nice to have all of the most popular platforms using the same file system for a change?
      BillDem
  • This makes everything so easy, now.

    If MS doesn't deliver it, then it's not really a necessary technology
    or all that important.

    It makes life so much easier when every grape you can't have is
    sour.
    frgough
    • Then by all means explain what benefit it brings.

      As it is you didn't answer the question.
      ye
      • Gee, I don't know

        running SQL queries at the file system level maybe? You know,
        future-vision kind of stuff.

        Here's a question back at you:

        If it provides NO benefit at all, then why is MS still pouring money
        into it?
        frgough
        • Nice in theory, bad in practice

          A lot of things that sound good i theory don't work well in the real world. Microsoft is still investing in these technologies for use with SQL Server, and some of the results might end up in a future operating system that doesn't have compatibility baggage to carry around. But for Windows, it was a big idea that had very bad consequences when they tried to implement it.
          Ed Bott
          • Agreed,

            but that's a completely different argument than "it brings nothing
            worthwhile to Windows," now, isn't it?
            frgough
          • Who said that?

            I said, "The reason WinFS was cut is that it didn?t actually solve a user need, it didn?t offer any compelling benefit to developers, and it dragged down overall system performance."

            I obviously disagree with your attempt to paraphrase that.
            Ed Bott
          • Do you read the stuff you write?

            "And they can continue to wait. The reason WinFS was cut is that
            it didn?t actually solve a user need, it didn?t offer any compelling
            benefit to developers, and it dragged down overall system
            performance."

            I would say my paraphrase is pretty accurate, but we can go with
            your actual quote if you prefer.
            frgough
          • "very bad consequences" ...

            ... that were what, insurmountable, or just a case of reaching too far in the given timeline?

            The last post on the WinFS blog had this to say (admittedly, written over two years ago):

            "Was WinFS "killed" because of its design?
            No. In fact, the Beta was coming together really well. People have speculated on "redesigns." The original goals of WinFS have never changed, but the technology we are building isn?t easy ? so we did take a number of internal design changes and re-writes. And I am not going to apologize for that. Getting the relational engine to behave and perform like the Windows filesystem isn?t a matter of a few lines of code ? it has to be done very carefully and architected right. The bars on performance, compatibility, etc. are all super high."

            http://blogs.msdn.com/winfs/
            Adrian Kingsley-Hughes
          • You believe that?

            That was the last public post by a guy whose project had failed and was about to be killed. Of course he was going to say everything was under control...

            I'm more willing to believe the comments from senior developers who were outside the WinFS project. Like the ones I quoted here.
            Ed Bott
          • Dunno ...

            ... I'm not going to get caught up questioning the honesty of the guy. It's on MSDN so I assume that if it was a total load of BS that Microsoft would have made it vanish by now ...

            ... but then again this is one of the reasons why some people have a hard time believing anything Microsoft says ... the message is sometimes all over the place. At least with Windows 7 we're not seeing that this time around.
            Adrian Kingsley-Hughes
          • A 2008 update

            So you believe the guy who has a powerful incentive to save face by publicly stating his product wasn't in trouble, right before his project is terminated, he disappears, and his blog went dark.

            But you won't believe Dare Obasanjo, who was writing his criticisms in real time and is still blogging regularly?

            Ther's a nice May 2008 interview with Quentin Clark, in which he acknolwedges that things really weren't going as well as all that:

            "That's kind of where we got tripped up in the Longhorn cycle. We were building too much of the house at once. We had guys working on the roof while we were still pouring concrete for the foundation."

            He says most of the WinFS work they did is now in SQL Server and will make its way into cloud-based systems for Microsoft.

            http://perspectives.on10.net/blogs/jonudell/Where-is-WinFS-now/

            I've updated the post with a new ending reflecting this stuff.
            Ed Bott
          • @Ed ...

            "So you believe the guy who has a powerful incentive to save face by publicly stating his product wasn't in trouble, right before his project is terminated, he disappears, and his blog went dark."

            No, I don't think I said that ...

            "But you won't believe Dare Obasanjo, who was writing his criticisms in real time and is still blogging regularly?"

            Or that ... all I said was that I'm not in a position to question the guy's honesty.

            But onto something Obasanjo did say (and I'm catching up on his writing on this subject right now) in the quote you pulled from 2004 - don't those performance and security arguments, along with the worry that the "programming model will suck" feel somewhat hollow to you?

            I'm not going to suggest that developing something like WinFS would be trivial or that it wouldn't be challenging, but that's different to saying that it had no merit.
            Adrian Kingsley-Hughes
          • Read the interview with Quentin Clark

            He says it better than I can. Putting it in the file system sounds like a good idea, but doing it turned out ot be a nightmare. Four years later they finally have a component version of it ready to ship. It will enable the app support that was the primary reason for WinFS in the first place. They are still years away from being able to confidently embed it in the file system.
            Ed Bott
  • I'll give you one reason right off the top of my head why I like WinFS ...

    ... and no, I've never said or hinted in any way that this was a replacement for NTFS.

    The aspect of WinFS I liked was the ability to create schemas that could be used to teach the system different file formats not already understood by the system. So, for example, a schema could be produced to allow indexing of, say, BlogJet files or a directory listing of say 7z files or an ISO file. This would have allowed third parties to extend the quality of search results beyond the limits placed upon it by Microsoft. I'm not currently aware of any system whereby Windows Search can be augmented.

    WinFS also, at least on paper, promised a far more organic search experience that didn't rely on keywords of knowing what file extensions you were dealing with.

    Another reason that WinFS was promising was that it was more open than what we currently have. The situation as we have it now is that Google Desktop Search beats Windows Search in both speed of indexing, processing speed and the quality of results on every system that I've tried it on. Unfortunately it doesn't work on 64-bit (or at least I've not tried it as the download claims to be 32-bit only).
    Adrian Kingsley-Hughes
    • Windows Search is Extensible

      You can extend Windows Search to index additional types of files.

      http://msdn.microsoft.com/en-us/library/bb231248(VS.85).aspx

      It appears this site has a lot of them: http://ifilter.org/
      MarkLR
      • Cheers for that ...

        Thanks
        Adrian Kingsley-Hughes
    • Go read the SDK

      Adrian, you say you're "not currently aware of any system whereby Windows Search can be augmented."

      Go read the SDK. Anyone can write an iFilter that expands the capabilities of Windows Search. Adobe and Foxit did it for PDFs. Office 2007 adds iFilters for its file formats. StarOffice has one for its files. There are in fact entire repositories of iFilters. More here:

      http://blogs.zdnet.com/Bott/?p=479&page=3

      You're asking to build an entire database on top of the file system, which app developers have to support, in order to get slightly faster search? Dmitry could write a BlogJet iFilter much easier than he could redesign the entire app to be compatible with WinFS. And that iFilter would allow BlogJet posts to work with SharePoint or IIS or any other product that uses Microsoft Search and Indexing services.

      As you note in your reply, WinFS promised more "on paper." But not in reality.
      Ed Bott
      • I'm gonna take a couple of these for a spin ...

        ... and see how well they work with natural language search.

        But this doesn't address the organic search issues and that there's no way to build relationships into the search.

        Another issue with the current search is false positives. Email searches are particularly bad at turning up stuff that's been moved or no longer exists ... and my systems claim that indexing is up to date. One would hope that a future system along the lines of WinFS would be transactional to a degree and know that a file had been deleted without the need to index.

        Do you have any numbers to back up that WinFS would only be "slightly faster" what what we have now?
        Adrian Kingsley-Hughes