How to avoid a FAT nightmare when upgrading to Vista

What you're about to read is a recantation of the trials and tribulations that ZDNet reader Tim Patterson experienced when he attempted to upgrade a Windows XP-based system -- one with the Windows Vista Capable sticker on it -- to Windows Vista. Although he eventually achieved his goal, things didn't go as swimmingly as he expected. It's an experience that begs three questions.

First, the question of how far Microsoft should be going to keep users from exiting what is otherwise a very user friendly installation script in order to manually execute a series of remedial steps that were prerequisite to getting back "on-script." Put another way, what contingencies should Microsoft's upgrade scripts account for, and what other ones must Microsoft accept as so-called collateral damage -- the point at which the risk of end-user frustration goes up and the cost to Microsoft of supporting those users by phone escalates to the point that money is lost on a copy of Windows Vista, rather than made. 

Second, the question of whether Microsoft's definition of "Windows Vista Capable" is different from what customers perceive it to mean. And third, what if anything can be learned from Patterson's experience so that others like him are best prepared for a smooth upgrade to Windows Vista.

Patterson isn't unlike a lot of other people who, when it came time to purchase a new computer last fall, decided to limit his search to computers that were preloaded with Windows XP SP2, but that were also capable of running Microsoft's forthcoming version of Windows (Vista). So, last October, he began his search and settled on an Acer notebook that had the "Windows Vista Capable" sticker on it. The question is whether what happened next when he finally got the opportunity to update to the Home Basic edition of Windows Vista is what awaits the buyers of others Windows Vista Capable systems.

Patterson's episode with the Windows Vista started innocently enough. He purchased the retail upgrade version of Windows Vista Home Basic which theoretically requires the pre-existence of a valid copy of Windows on the system in question in order to work (Ed Bott has a work around discussed later). It was here that Patterson hit his first speed bump:

I attempted to install the upgrade from within Windows XP [per the instructions]. The install progressed through the dialog which had me enter the [key for Windows Vista]. Then, the Vista install presented me with a list of partitions and asked me to choose the partition containing the upgrade target (Windows XP). I chose the proper partition (C:) and [the installation routine] displayed an error dialog informing me that Vista needed an NTFS partition in order to perform the upgrade.

NTFS stands for NT File System. For all intents and purposes, a filesystem like NTFS is responsible for the specific ways in which an operating system like Windows saves and retrieves information to and from a computer's hard drive. Most users of Windows don't know or care about the specifics of its underlying filesystem, nor should they. But, as you can see, Patterson found out the hard way that the presence, or lack thereof, of an NTFS-based filesystem under his existing Windows XP installation was going to make or break his chances of upgrading to Windows Vista.

The problem? As it turns out, the version of Windows XP that came pre-installed on the Acer notebook had a filesystem underneath it; it just wasn't the NTFS filesystem. Instead, it was the 32-bit version of NTFS' predecessor FAT (a.k.a. FAT32). FAT stands for file allocation table and dates back to the days of MS-DOS (Microsoft Disk Operating System). Whereas the 32-bit versions of Windows that came before Vista (including Windows XP) were all capable of running on FAT32-formatted partitions (one or more partitions must be created on a hard drive before a filesystem can be installed on it), Windows Vista is not. To the extent that filesystems and partitions are operating system underpinnings that most users have traditionally been sheltered from, is requiring awareness of filesystem specifics, particularly from non-technical users looking to do little more than plug an upgrade disk in and accept the defaults, a bit of a calculated risk on Microsoft's behalf? I'll get to the answer in a moment.

But first, here's what happened next:

The dialog then says that I must boot [from the Windows Vista installation DVD] and create an NTFS partition. Upon booting from the upgrade DVD and prior to getting any option to format or convert [FAT32] partitions, I got an error message telling me that the Vista upgrade must be installed from within XP. This is a dead end.

In other words, if a utility to create an NTFS partition (or convert the existing FAT32 one) exists on the Windows Vista installation DVD, booting up the computer from that media (as Patterson was instructed to do) doesn't appear to be of much help in getting there. Users that go down the same path as Patterson will bump into an error dialog instructing them to reload the upgrade routine from within Windows XP, putting them right back where they started (on the first path to the error dialog about FAT32).

Patterson's problem, as it turns out, is solvable. It's just that the solution wasn't staring Patterson in the face from his computer or from the package of materials that came with the Vista upgrade DVD. Patterson contacted Microsoft's technical support who decided to draw Acer's technical support onto the call, but not before advising Patterson that he would need to convert his FAT32 partition to NTFS. He was also advised that such conversions can result in corruption of his data and applications and was advised to make a backup of everything before doing anything. In anticipation of a long phone call with Microsoft and Acer, Patterson hung the phone up and started to research the problem on his own (a process that resulted in his first note to me asking what I thought he should do next). Eventually, he ended up running a command line utility from within XP that's specifically for converting an existing partition to NTFS:

convert C: /fs:ntfs where "C:" represents the drive letter of the partition to convert.

After completing the conversion (which resulted in no data or application corruption), Patterson says things went pretty swimmingly once he went back on script. But somewhere along the line, his entire previous Windows XP installation ended up in a directory called Windows.Old -- a shuffle of the bits that disabled his previously installed Windows applications. Patterson, like many other Windows users, had applications on his hard drive that involved the downloading of software and online purchasing of license keys. I myself have been caught in similar traps where, even if I know what the old license key is to some software I downloaded in the past (many people don't keep this information), I often find that when I return to the developer's site, the only version of the software that's available for download is a newer one that may not work with my old license key.

It's not that I don't trust Patterson. His notes were very sincere and the email thread actually started with a request for advice on what to do next when he was stuck. But I've been duped by readers with an agenda before. So I did some Web searching and then I double checked with our resident Windows guru Ed Bott to see if any of this rung a bell with him. Not only did I learn that Patterson wasn't alone in his experience, I have the answers to the the "off-script" and "Windows Vista Capable" questions at the top of this blog and I have some advice for anybody who is upgrading to Vista, not necessarily just those who are upgrading from a FAT32-based filesystem.

Some systems do come with FAT32 pre-installed: Although he seems a bit confounded by the system manufacturer's choice, Bott confirmed that Acer ships some models of its Windows XP-based computers pre-formatted for FAT32 instead of NTFS. According to Bott, his own Acer-made tablet notebook came this way as well and the 100 GB hard drive that came in it was divided into two 50 GB FAT32-based partitions. I did a quick search to see if any other system manufacturers were doing the same thing and stumbled upon a message on one of CNET's message boards indicating that Systemax was doing the same thing in its Venture VX2 systems. Why would system makers do this? Performance might be one reason. FAT32-based systems have outperformed NTFS-based systems in some benchmarks. (Update: this ZDNet reader has another explanation)

Advice: It's hard to know which systems will be pre-formatted for FAT32 vs. NTFS before you buy them. It's generally not a detail that's listed on the side of the box. But, if you've recently purchased a supposedly "Windows Vista Capable" system or plan to buy one with the intent of upgrading it to Windows Vista, double checking the filesystem type and converting it to NTFS (if it isn't already NTFS) is something to do sooner rather than later. In fact, right out of the box if possible. Why? Because if some corruption does result from the conversion, there will be less to reconstruct.

Also, while you should always have a recent enough backup of your system in order to reconstruct it anyway, it's doubly important to have such a backup before converting a system's filesystem. This is true even if you just unboxed the computer since most computers these days don't even come with the original operating system installation CDs from Microsoft. Instead, many systems (including Patterson's Acer) come with a utility that generates "recovery" CDs: bootable CDs that can automatically restore a system to the last "backed-up" state or that can restore specific files.

"Windows Vista Capable" is confusing:  According to Bott, the "Windows Vista Capable" emblem is about the capacity of the hardware to handle the load of Windows Vista (and all its features) and not about how the system comes configured. In fact, Microsoft's own description of the Windows Vista Capable program makes no mention of a filesystem predisposition. Bott says that Patterson's system undoubtedly satisfied Microsoft's hardware requirements in terms of being able to run Windows Vista. But the choice by a system manufacturer to format a system's partitions with FAT32 instead of the preferred-for-Vista NTFS filesystem is irrelevant to Microsoft's "Windows Vista Capable" program.

Advice to Microsoft: Keeping Vista-upgraders "on-script" is not just in the end-users' best interests. It's also in Microsoft's best interests as well as those of the system manufacturer. Regardless of how Microsoft defines "Windows Vista Capable," I see it as meaning that the system in question can be easily upgraded to Windows Vista. Once users must go off-script to complete an installation or upgrade, not only is "easily" out the window, the chances that those users will need technical support from Microsoft, the system maker, or both go way up. In judging their systems to not be Vista-capable, certain customers will be less satsified with their experience, with Microsoft, and with the system manufacturer. Microsoft should insist that "Windows Vista Capable" systems be formatted at the factory with NTFS rather than FAT32.

Advice to End-Users: Until Microsoft officially changes what's required of system manufacturers before they can affix Windows Vista Capable-emblems to their PCs, buyers of such PCs should consider the aforementioned advice to double check what filesystem comes with their PC and to convert it sooner rather than later (but not before backing it up).

Should you buy the retail upgrade to Windows Vista: As with the newly released versions of any of Microsoft's operating systems, there is always a class of early adopters and enthusiasts who are chomping at the bit to experience the new code. In the case of systems purchased between October 26, 2006 through March 15, 2007, many buyers will find an Express Upgrade coupon that entitles them to a discounted upgrade to Windows Vista from the system manufacturer. In some cases, the upgrade is free (not including the $10 shipping and handling charge) and others not. But the key here, according to Bott, is that none of the system manufacturers are shipping their upgrade disks yet even though it's possible for customers to order them.

In other words, for those early adopters and enthusiasts, the only way to acquire a Windows Vista upgrade is to forego the discount and purchase a retail version at a local store or over the Web. One question, Bott asks, is whether or not the system manufacturers that are shipping XP-based systems configured for FAT32 instead of NTFS will account for the need to do a filesystem conversion in the upgrade media they ship. Whereas customers purchasing the retail version have no choice but to go "off-script" to complete their upgrade, will system manufacturers like Acer have a different script that automates the process? Said Bott, "We won't know until the disks come out."

System buying advice: For starters, if you're still thinking about buying a system that's pre-loaded with Windows XP (soon to be a very rare find anyway), Bott suggests that you not buy a system that's pre-loaded with XP Home. The reason is that, for most vendors, even with the Express Upgrade discount, upgrading to the Home Basic edition of Windows Vista (which is all XP Home can upgrade to with the coupon) will cost $55 (including shipping and handling). Meanwhile, Express Upgrading to the Home Premium edition of Windows Vista (as long as you currently have the Media Center version of Windows XP) is usually free (plus $10 shipping and handling). 

So, the better choice when buying a new XP system is to pay the extra $29 that vendors like Dell charge for the Media Center version of Windows XP in order to entitle yourself to a free upgrade to Windows Vista Home Premium. Total out of pocket cost? $39.  Not only do you get the more robust version of Windows Vista (normally $160 at CompUSA), you pay $16 less for it.

Upgrading advice: Upgrade from your system manufacturer, even if it involves a bit of a wait, if you can. First, even if we're talking about the $55 upgrade for Windows Vista Home Basic, that still represents a $45 savings over the $100 street price for the retail version of the same thing. If you paid $29 to upgrade to Windows XP Media Center and wait for the free upgrade to Windows Vista Home Premium (a $160 value), you just saved $131. Another reason to wait is that if there's something very specific to the system that retail upgrade doesn't account for -- be it something related to the filesystem (FAT32 vs. NTFS), or maybe special hardware drivers, or who knows -- there's a chance that the upgrade media from your system manufacturer will account for those system-specific items in their upgrade scripts, thereby offering the smoothest upgrade experience that allows you to stay on-script.

Should the Vista Upgrade have included a FAT32 to NTFS conversion step: This was actually the first question that entered my mind. How on earth could the Windows Vista upgrade not account for this contingency? Not only are new systems bearing the "Windows Vista Capable" sticker still entering the market with FAT32-formatted partitions on them, but, if you look at the basic hardware requirements in order for a PC to qualify as Windows Vista Capable (an 800 Mhz processor, 512MB of memory, and a graphics processor that supports DirectX 9), that could account for a lot of PCs that are already in the marketplace, many of which could be formatted for FAT32.

Thoughts: First, judging by an entry in the company's online knowledge-base, Microsoft is aware that the problem exists.I can understand why Microsoft doesn't include a FAT32 to NTFS conversion as a standard part of the upgrade process. When Patterson called technical support, the person on the other end of the phone insisted on walking him through a complete backup of his system before moving onto the step of conversion. While Ed Bott told me he has never heard of corruption resulting from such a conversion, the chance of corruption exists and Microsoft would be crucified for including a potentially-data-corrupting step in its upgrade process that it could have otherwise excluded.

The options to Microsoft, from a user-experience point of view, if it did include such a conversion step from within the script, would be to either (a) plaster the equivalent of a red neon sign in front of users warning them to backup their data first (and that Microsoft can accept no liability for data corruption, or (b) offer users a way to back their installation up from within the script. While system manufacturers may end up offering such an automated step in their upgrade media that's tied into whatever backup solution they build into their systems (like IBM's Rescue and Recovery Utility, these differ from manufacturer to manufacturer) -- I think it makes less sense for Microsoft to build a full PC backup operation built into what is otherwise an upgrade process. For example, whereas vendors may have an idea of what the removable media capabilities (necessary for backing up) are on the PC's they've sold, Microsoft really has no way of figuring that out without really taking end-users off-script.

• Advice: Upgrades should have nothing to do with backing up your data. A lot of people growl at the idea of having to go through a time consuming backup as a prerequisite to some sort of time consuming technical operation that could put your data at risk. The bottom line is that your data is ALWAYS at risk and you should be backing up all the time anyway. If you don't have a backup and want to be mad at someone for making you do one (particularly a time-consuming one), be mad at yourself.

The different types of Vista upgrades: So, let's say you've reached the point that Tim Patterson has reached. Not only is it time to upgrade, but a filesystem conversion is required in order to proceed. According to Ed, there are actually four separate choices you can make when it comes to the type of upgrade you can do:

• The first of these is an in-place upgrade. This is where, by the time you're finished, your applications are still in their designated directories and the upgrade process has done the best job it could possibly do in preserving their integrity. Things like User IDs for system acces and bookmarks for Internet Explorer are saved.The hope is, of course, that whatever applications you were using before the upgrade will work pretty much the same way after the upgrade.
  
• The second type of installation is the one where the integrity of your existing implementation isn't preserved but instead of wiping out your data and applications, your entire old XP installation is tucked into a directory called "windows.old" and, other than that directory, your installation of Vista is a pretty clean slate. User IDs are not preserved, nor are other personalizations like bookmarks for Internet Explorer. This was the path that our reader Tim Patterson took.
  
• The third type of installation is a variation on the second where the old installation is preserved. The difference is that instead of having a windows.old directory, the previous installation is preserved exactly as it was on its partition and Windows Vista installs itself onto a separate paritition on your system. According to Ed Bott, when you're finished with this type of upgrade, you essentially end up with a dual boot system.
• The fourth type is neither published or supported by Microsoft. As it turns out, the DVD media used for the full-blown versions of Vista and the upgrade versions are exactly the same. What's different is the license key. If you have an upgrade license key, the installation process tries to, as best it can, validate that you have an existing and valid copy of Windows on your system already. One type of upgrade installation that some people want to do is a completely clean installation. In other words, wipe the sysem completely clean of any vestiges of the old system so that when the upgrade is finished, it's like a brand new system with Vista. But there's one catch. In order for an upgrade license key to work, you can't boot from the DVD. The DVDs have to be run from within the existing copy of Windows (as noted in Patterson's case). For those who want a like-new system, if you're running the upgrade routine from within Windows XP, there's no way to wipe that copy of Windows XP out because the routine needs that copy of Windows to complete its work. But as it turns out, there's a workaround that Ed Bott will be publishing tomorrow.

With these four upgrade types in mind, what should you do?  As we learned from Patterson's case, there's a bit of a catch-22 in deciding which of these paths to take. Knowing how Windows can easily get cluttered up over time, the preference is always to start from a clean slate (essentially, one of the last three options above). But that could present a problem if you're one of the many users that has acquired try-and-buy applications and the keys to unlock them through the Internet.

For example, I have a lot of software that started with a 30-day trial period, but then involved the online purchase of a key to use it for longer. This is very different from buying software in a store where you get a box, some discs, and a printed version of the license key. Somewhere along the line, I always seem to lose the two most important things in order for me to be able to reload that software onto a new system: the original download (the one that the license key works with since the newer versions of the software usually don't work with old keys without paying some upgrade fee) and the license key itself. Some developers of try-and-buy applications offer means for recovering this information and I've even heard of third party Web sites that, for a fee will keep this information for you in an online locker of sorts (and some developers actually interface with these services automatically). But reconstructing all of those packages (of the original downloads and the license keys) can be a nightmare.

Advice on Try-and-Buy software: Today, I'm a bit more organized than I used to be when it comes to my software. In anticipation of the fact that I may have to reinstall try-and-buy software that I bought, I basically recreate my own package of the software as though I purchased it in the store. I take a blank CD and burn two things to it (1) whatever installation package I downloaded to my hard drive (2) a text file to which I cut-and-pasted every relevant piece of information that matters. For example, license key numbers, user IDs, which of my many e-mail addresses I used to register the software, copies of email receipts, etc. 

Then, I go one step further. On that CD, I take a sharpie, I write the name of the software on the CD as well as the license-key for that software. Not only do I do this last step for my try-and-buy software, I do it for any software where the discs came from a store or the developer. This way, if the CDs or DVDs ever get separated from their packaging (packaging that may have the license information printed on or in it), I still have the license information right there on the CD.

Advice on which of the four upgrade paths to take: When I asked Ed Bott about this, his suggestion (and I agree) is to go for one of the clean installations. But, in addition to any collateral damage you may have to take on (eg: reconstructing your application setup, including try-and-buy apps) in order to get a clean install, there are other ideas and things to consider:

Before upgrading an operating system, one of the first things I like to do if I'm not using upgrade media that came from my system's manufacturer is, with my still functioning system, go to that manufacturer's Web site and download any drivers for the new operating system that it may offer and then I burn those drivers to CD-ROM.

With the second of the four upgrade options (the one that generates a windows.old directory), you'll need to make sure there's actually enough space on your hard drive to accomodate a repackaging of your old setup and the 11 gigabytes of data required to install Windows Vista. According to Bott, the Vista upgrade will figure out if you have enough room before attempting to officially start. If you don't, then you should look into what it is that's consuming so much of your existing disk space and find an alternative home for it, perhaps on a USB-based hard drive that you can buy at your local Staples.

The third option may, from a storage perspective, present the same challenges as the second in that there isn't a whole lot of difference between putting Windows Vista on the same partition that your existing Windows installation is on vs. another one on the same hard drive. In other words, does your hard drive have the room to create another partition and put Vista on it? Only you know the answer to this question. 

The fourth option -- the one that Microsoft doesn't publish or support is probably the best. Not only does it give you a pristine clean install, but it actually accomodates that one tricky situation where you may decide to upgrade your hard drive at the same time. Particulary with notebooks where, when you purchased your laptop, the biggest drive that could fit into it might have been 40 or 80GB. But now, the odds are pretty good that you can get a 100 or 120-gigger into it.

Tomorrow, over on Ed Bott's blog, he'll be publishing a walk through of how to excercise the fourth option. It's part of a 30-tip series he's developing for people diving into Windows Vista now.

If Microsoft's explanation of how to convert from FAT32 to NTFS isn't good enough for you, check out Ed Bott's shorter explanation of how to accomplish the same thing. One other tip going back to Tim Patterson's catch-22: If you boot your computer from the Vista upgrade DVDs, it's actually possible to reach the partitioning utilities on those DVDs without bumping into an error dialog (telling you to reload from within Windows XP) first. The trick? When it comes time to enter a license key, don't enter one. A Vista installation will actually function for 30 days without a license key. But there's a couple caveats to this approach. First, the only partitioning utility you can reach (as a part of the process of doing a fresh installation) is the one that deletes existing partitions or creates new ones. It doesn't offer the ability to convert existing ones. Second, by the time you're done installing Vista, the only type of license key it will accept (in order to use Vista beyond the 30 days) is one for the full-blow version (not an upgrade). But this is where Ed's workaround takes over. Stay tuned to his blog tomorrow. 

Update 2/15/07: Given the number of comments about who is at fault here, I posted a follow up to this blog that discusses the semantics of "capable" vs. "ready."  Microsoft used the phrase "Windows XP Ready" when it launched XP back in 2001. Also, on that follow up is a poll where you can vote to indicate which party is most at fault here.