Why do you want WinFS?

Every time the conversation turns to Windows 7, the subject of WinFS comes up. This next-generation file storage software was cut from Windows Vista (and eventually killed completely) nearly four years ago. But it keeps reappearing on lists of features that people want to see in Windows 7. Why? WinFS made for great PowerPoint slides, but it was a terrible idea. Don't just take my word for it: Some of Microsoft's most experienced and respected developers agree.

One point I made in my wish list for Windows 7 deserves repeating:

Several commenters on [Sinofsky’s] initial “Welcome” post expressed hope that the WinFS file system, which was killed off during the infamous “Longhorn reset,” would be resurrected for Windows 7. Not gonna happen. Nor, frankly, is it necessary.

Last time I looked, at least four commenters on Sinofsky's post had pleaded for the return of WinFS. Within a few hours of my post, my ZDNet colleague Adrian Kingsley-Hughes banged out his Windows 7 wishes. Right smack in the middle of the list was “WinFS …” No further details, just the bare item. In the Talkback section of Adrian’s post, I was intrigued to see this comment from Larry Osterman:

Why do you want WinFS?

What [capabilities] did WinFS give you that Vista doesn't already provide?

Good questions. And the guy doing the asking isn't just some random passer-by, either. Larry Osterman is a veteran Microsoft developer (started in 1984) who has written code for every member of the Windows NT family and who currently works on the core Windows audio engine. (His blog is a must-read if you’re even slightly interested in the Windows Vista audio stack.) Every time I hear someone pining for the return of WinFS, I ask the same question as Larry. Why do you want WinFS? What problem are you trying to solve? Although it made for great PowerPoint slides, WinFS was a terrible idea, and killing it was one of the smartest things Microsoft ever did.

For those who need a little background, WinFS is short for Windows Future Storage. (Wikipedia has a long, dry, technical overview that’s worth reading if you want more details. NTFS.com has a slightly more accessible explanation, complete with diagrams.) It was one of the “gee golly whiz” features of the Longhorn project, which was the original codename for the operating system that became Windows Vista. WinFS was announced at the 2003 Professional Developers Conference (PDC) and actually shipped in a very early beta of Longhorn, back in 2004. Unfortunately, it didn’t work particularly well, an attribute it had in common with a lot of other stuff in those early betas. A few months after that 2004 beta release, the Windows team reviewed the grandiose plans for Longhorn and scaled it back, a process that later became known as the “Longhorn reset.” WinFS was one of the first things cut and it was later killed as an OS component. (If you want to see the hype cycle for this feature in its full splendor, go read the WinFS Team Blog from start to its ignominious finish in June 2006, just before Vista was locked down.)

WinFS is probably the most misunderstood Windows feature of all time. The biggest misconception is that it was going to replace NTFS, the file system that’s used by every member of the Windows NT family, which includes Windows Vista. (The fact that each name ends in FS makes the confusion almost inevitable.) Although it looks and acts like a file system, it isn't; WinFS was intended as a SQL database that sits on top of the underlying file system (which means you could replace NTFS with a different file system or port WinFS to another operating system and it would still work). The idea was to allow applications to share, compare, and aggregate data from different stores.  I saw it demoed several times, publicly and privately, back in 2003 and 2004. All the capabilities I was shown then are in Vista today and will be in Windows 7 when it arrives, using XML, tags, and the systemwide index provided by Windows Search.

Way back in 2004, I wrote about this decision in a post entitled “Longhorn wish list.” (Ironic, eh?) My take at the time:

Honestly, I don’t care that much about WinFS and Avalon. They’re cool, but they can wait.

And they can continue to wait. The reason WinFS was cut is that it didn’t actually solve a user need, it didn’t offer any compelling benefit to developers, and it dragged down overall system performance. Back in June 2004, Microsoft’s Dare Obasanjo nailed this point [emphasis added]:

[T]he core scenarios touted for the encouraging the creation of WinFS (i.e search and adding metadata to files) don't really need a solution as complex or as intrusive to the operating system as WinFS. The only justification for something as radical and complex as WinFS is if Windows application developers end up utilizing it to meet their needs. However as an application developer on the Windows platform I primarily worry about three major aspects of WinFS. The first is performance, I definitely think having a query language over an optimized store in the file system is all good but wouldn't use it if the performance wasn't up to snuff. Secondly I worry about security, Longhorn evangelists like talking up what a wonderful world it would be if all my apps could share their data but ignore the fact that in reality this can lead to disasters. Having multiple applications share the same data store where one badly written application can corrupt the entire store is worrisome. This is the fundamental problem with the Windows registry and to a lesser extent the cause of DLL hell in Windows. The third thing I worry about is that the programming model will suck. An easy to use programming model often trumps almost any problem. Developers prefer building distributed applications using XML Web Services in .NET to the alternatives even though in some cases this choice leads to lower performance. The same developers would rather store information in the registry than come up with a robust alternative on their own because the programming model for the registry is fairly straightforward.

It’s worth noting that Dare wrote that when WinFS was alive and still being vigorously defended. That took some guts. A few years later, another Softie, Matt Warren, called WinFS “the biggest fiasco [he] ever bore witness to.” Back in 2003, he was working on an alternative project called ObjectSpaces. Here’s how he remembers WinFS:

The WinFS client API even started out as a complete copy of the ObjectSpaces codebase and had all the same limitations. It just had more political clout as it was being [led] by a figure at a higher-point in the corporate org chart, and so it was positioned as part of a juggernaut that was making a massive internal land grab. We on the outside used to refer to WinFS as the black hole, forever growing, sucking up all available resources, letting nothing escape and in the end producing nothing except possibly a gateway to an alternate reality. Many of our friends and co-workers had already been sucked in, and the weekly reports and horror stories were not for the weak-of-heart.


There were not too many in the developer division that believed in the mission of WinFS. As a developer tool for the masses, something simple that targeted the lower end was paramount. ObjectSpaces had been it, and now it was gone. There was still some glimmer of possibility that WinFS v2 might eventually get it right and be useful as a general ORM tool. But all hope of that was shot when WinFS was pulled out of Vista and its entire existence was put in doubt.

The next time someone tells you that WinFS is on their wish list for Windows, do me a favor and send them here. I’m still trying to find someone who can give me even one good reason.

Update: After posting this, I ran across an excellent May 2008 interview that Jon Udell did with Quentin Clark, who ran the WinFS project, and who posted much of the content on the WinFS Blog that I cite earlier. The interview is entitled "Where is WinFS now?" and in it, Clark acknowledges that the Longhorn effort was not the epitome of smooth development: "[W]e got tripped up in the Longhorn cycle. We were building too much of the house at once. We had guys working on the roof while we were still pouring concrete for the foundation. At one point we realized we needed to decouple things. And that really did give this team the freedom to go off and take these underlying technologies, which we believe were fundamental to the database, and get them done correctly."

He also confirms my suggestion that the best home for these technologies, at least for now, is not in the file system but rather in the cloud computing environment. The big news is that the follow-up to WinFS is "shipping in SQL Server 2008, as a feature called filestream." So it will be an add-on that apps can take advantage of, but it is not ready to integrate into the file system today:

Anyway, we did a partnership with Microsoft research, and at some point along the arc we solved it fairly well. It's not trivial. This is not something that ends up being a simple solution to this very complex problem. It's actually reasonably sophisticated, but it works, and we built it in as part of the last WinFS beta.

As they realized they were onto something, they started to fork out a componentized version of it that's now finding its way into a bunch of Microsoft products. The official branding is Microsoft Sync Framework. I think they're on target for shipping it in six different products, and for embedding it all over the place.

If you're interested in this stuff, the whole interview is well worth reading.