Are Microsoft's update servers broken?

Update 14-Aug: I've posted a follow-up here with a few more details.

On Tuesday, August 8, Microsoft released a total of 12 security updates for different versions of Windows, Office, and Internet Explorer. One, MS06–040, is a vulnerability in the Windows Server service and affects all versions of Windows XP. The latter is so serious, in fact, that the United States Department of Homeland Security issued an urgent news release recommending that all Windows users apply this security patch as soon as possible. At least seven other updates on this month’s list are rated Critical for users of Windows XP with Service Pack 2 and involve the potential for “an attacker to compromise your Windows-based system.”

All those patches are supposed to be released via Automatic Updates and automatically delivered This week, Microsoft's update servers have apparently collapsed under the load to computers that have the Automatic Updates service turned on. They’re also available via Windows Update. Millions of people rely on these servers to keep their Windows computers secure.

But this week, Microsoft’s update servers have apparently collapsed under the load. In my office, I have four computers running various flavors of Windows XP. All have Automatic Updates enabled. Four full days after the release of these Critical updates, only one of those machines is fully patched. It turns out I’m not the only person experiencing problems. A search of public newsgroups turns up dozens of complaints about Windows Update and Automatic Updates not working properly:

August 9: When running Window update it hangs up during it's search for updates every time I try it. The only way out is to power down. Task manager shows the cpu usage at 100%.

August 9: I've never had any issues w/Windows or Microsoft Update until today's patch release. Some installed, but KB920214, KB920683, KB921398, and KB922616 will not install either via Automatic Updates, or by me downloading the file from MS Downloads and running it myself. Each of them fails w/the same sequence of errors…

August 9: I have my updates set to notify me when there is an update. I tried to downloand them yesterday, 8/8/06, and they would not install. When I tried to access them via the windows update site...it just kept seaching for updates...then froze... I spent 2 hrs..on the phone with microsoft...clearly there is a problem they are not yet aware of.

August 10: I'm having the same problem with the August updates. This never happened before. It happens both with automatic updates, and by selecting Tools -> Windows Update from the IE menu. My resources get so taxed I can't even pull up task manager or restart, I literally have to unplug the computer! I'm running Windows xp sp2 and have made no software or setting changes since the July updates. I finally disabled automatic updates til MS acknowledges and fixes the problem.

August 11: Each time I've turned on the computer for the past few days a small yellow shield pops up in the system tray saying the updates are being downloaded. But the shield disappears after about 30 seconds and nothing is downloaded.

August 11: We have setup our computers for automatic Windows Updates. It seems as though none of these updates are actually running. There is an error in the event viewer … “Unable to Connect: Windows is unable to connect to the automatic updates service and therefore cannot download and install updates according to the set schedule. Windows will continue to try to establish a connection.”

I can confirm all of these experiences. On three of four XP-based computers here, the MS06–040 patch was properly installed, but no other updates were delivered. A check of the Windows Update History shows a gray Cancelled button next to one or more of the missing updates. The Windows Update log file (%windir%\Windowsupdate.log) on two computers shows dozens of attempts to download the other patches, followed by this error message: Update is not allowed to download due to regulation. Microsoft Knowledge Base article 910340 explains what this message means:

During periods of heavy download traffic, the Automatic Updates service can reschedule download requests on a day-to-day basis. This rescheduling can occur over several days.

I’ve looked in vain to find what Microsoft’s guarantee of service is on Automatic Updates. In the past, I’ve never had to wait more than 48 hours to receive updates. A four-day wait is simply inexcusable.

Update 12-August 4:30PM PDT: No, it's not my imagination. The performance of Automatic Updates has definitely slipped recently. I have one machine that has been in continuous operation and has been configured for Automatic Updates since late 2004 (that's when Microsoft inaugurated the "Patch Tuesday" program, in which Critical security updates are released on the second Tuesday of each month). By checking the Windows Update history on that machine, I was able to plot when the updates were actually delivered and compare the delivery dates to Patch Tuesday. In 11 of 13 months between December 2004 and January 2006, this machine received updates on the Wednesday or Thursday after Patch Tuesday. On average, it took 2.0 days for Automatic Updates to arrive in 2005. Since February 2006, those updates have routinely arrived 3-5 days after they were released, an average of 3.4 days later. To put it another way, only once in the 14 months between December 2004 and January 2006 did I have to wait four days for updates to arrive. Since the beginning of 2006, Automatic Updates have been delayed for at least four days in four of seven months for which I have data.

Ironically, the problem has also affected the computer I’ve been using to test Windows Genuine Advantage. On Thursday, I reinstalled a volume license version of Windows XP using a blocked product ID supplied to me by Microsoft; since then, I’ve been unable to download any updates, including the WGA validation and notification components. When accessing Windows Update, the system either hangs completely or displays an unhelpful error page (0x8DDD0009). At irregular intervals, Automatic Updates displays a yellow shield icon in the taskbar, which notes that downloads are updating and displays a progress meter reading 0%. It goes away after a few minutes without succssfully retrieving any updates. So this computer is blissfully free of WGA nag messages; unfortunately, it’s also free of crucial security updates.

On two “certified Genuine” computers, one running Windows XP Media Center Edition 2005 and another running Windows XP Home Edition, I was able to manually install all of the missing Critical updates by visiting either the Microsoft Update or Windows Update site. But having Automatic Updates turned on means I shouldn’t be required to do that.

A few weeks ago, Microsoft announced it was going to push IE7 as an automatic update when it’s finally released this fall. At the time, I thought that was a pretty good idea. Now I’m not so sure. If the infrastructure for delivering Critical security updates is failing under the current load, what impact will a big update like IE7 have?

Microsoft needs to explain exactly what’s going on here. What’s the problem with Windows Update, and when will it be fixed?