Why all the errors in Microsoft updates lately?

About a month ago I wrote a column celebrating the great things that Patch Tuesday has done for customers and the industry. I still believe in it, but I couldn't have picked a worse time to write it. In the weeks that followed, Microsoft customers have experienced a reign of error under Windows Update.

A few days after my column appeared, Microsoft was forced to withdraw two August patches, beginning with a patch for Outlook Web Access in Exchange Server. The buggy code in this patch turns out, ironically, to be written by Oracle, but that's neither here nor there: Microsoft delivered it as part of their product and it caused problems on Exchange Server 2013. The second patch they withdrew was for ADFS (Active Directory Federation Services), but they re-released it a few days later.

The Exchange Server update wasn't re-released until late in August, at which point they also re-released a separate July patch for Windows Media Services that had not been withdrawn.

Anyone can have one bad month I guess, but it didn't end there. Yesterday Microsoft pulled a buggy non-security update to Outlook 2013. They explained the problem and what was happening in a Technet blog entry, but it's still not over.

There was at least one more buggy patch in September, described in this support Microsoft forum and this Technet thread. The problem seems to be related to the patch for MS13-074, a security update for Access. I was a victim of this one. The first thing I saw was that I couldn't load any Office (2013) apps. I got the same unhelpful "something went wrong" error message.

The problem most users report is that, even after installing the patch, Windows Update reports that it is not installed. Even if you manually install the standalone version of the patch, which appears to install correctly, Windows Update still reports that you need to install it. Go to Programs and Features and look at the installed updates and you'll see the update there (designated by its KB number, KB2810009). You can uninstall it and try again, but it won't make a difference. I wasn't able to run Office programs again until I used System Restore to revert the system back to pre-Patch Tuesday. I haven't seen a response from Microsoft on this one.

Two bad months in a row? And not too long ago, in April, Microsoft had to call on Windows 7 users to uninstall an update that was crashing systems. This level of quality is atypical.

It's not that Microsoft doesn't care. They put tremendous resources into updating their software. I asked about this latest pattern and Dustin Childs, Group Manager, Microsoft Trustworthy Computing, replied: “The quality of security updates is critical to our customers, and it is a high priority for us too. We are actively looking at where improvements can be made with the goal of reducing implementation issues, and we will remain transparent with our customers about security threats, protections and update issue resolution.” Below this article is an embedded video about Microsoft's security updating process featuring Childs.

I went to do some research on previous problematic updates in the last several years. It's not easy. Based on my fishing expedition through Technet, I'd say that clusters of errors like this have been very rare in recent years. There was one in April 2011, detailed in these entries on  the Microsoft Office Updates blog ("The official blog of the Office Sustained Engineering and Release team"):

• Issues After Installing PowerPoint 2002 Update KB2464617 (30 Apr 2011)
• Issues after installing Outlook KB KB2509470 (28 Apr 2011)
• Issues after installing PowerPoint 2003 update KB2464588 (23 Apr 2011)

Without the updates and their versions in some structured database it's hard to go much further, and I haven't been able to get Microsoft to say any more on it. But I have a theory.

September was a particularly busy month for Microsoft patches. There were 13 security bulletins covering 47 vulnerabilities. In fact, there was a 14th bulletin that was withdrawn in the final days for QA reasons; this last point both adds to the concerns about quality and shows that Microsoft does put some effort into it and is willing to hold back on a patch.

But there were also numerous non-security updates; the details of all updates, security and non-security, for the year 2013 are in this support article. The list includes the monthly Windows Malicious Software Removal Tool.

Or is it really all the updates? The first botched update this month, the one that messed up Outlook 2013, isn't on the list of non-security updates, and yet it shows up in the list of installed updates under Programs and Features.

And there's even more than that. If you follow Patch Tuesday as closely as I have over the years, you notice that there is plenty of credit given to outside researchers for reporting vulnerabilities, but rarely if ever do they say that Microsoft found one internally. That's because those vulnerabilities are often patched silently.

The numbers are large not because of any general quality deficit at Microsoft, but because they have so many products. Microsoft is not the only company that patches software and not the only one that has had embarrassing outcomes from buggy patches. Firefox 16 had to be yanked off the download servers because of a really bad vulnerability. In 2010 McAfee released a virus definition update with a false positive so bad that it rendered Windows XP SP3 systems unusable. The company actually ended up paying for repairs to customer systems. Microsoft has never had anything go that wrong. But these things happen; software is really complex, nobody is perfect, and some percentage of the inevitable errors are inevitably bad ones.

I'm sure Microsoft puts enormous resources into keeping the update monster happy and well-fed, but perhaps it's just too big now. I see signs recently that the burden is too great. All of the buggy updates - and this is basically by definition - were insufficiently tested. I wouldn't be surprised if having all those products shooting for the 2nd Tuesday of the month was causing scheduling conflicts in testing and giving short shrift to some tests. I've done professional testing for a long time and I know that a proper test can be quite time consuming.

And perhaps one day a month is not often enough. Patch Tuesday was inaugurated as once/month because IT wanted to be able to plan to dedicate resources at a specific date and time. I suspect that the update machinery is well-enough defined that two per month would not be a great burden.

In fact, Microsoft already does have two Patch Tuesdays a month! The company also releases updates on the 4th Tuesday of the month, but only updates for performance, reliability and application compatibility. It's not a secret - that same support article with all the update definitions has the second Patch Tuesday updates in it too. So maybe even more are warranted, perhaps organized by product family. It's only the critical security vulnerabilities for which IT needs to be ready to move ASAP, and one of those a month is enough.

I'm guessing here, but I do think that it must be on senior management radar as a big problem to address. I have no doubt that they're getting lots of complaints, and not just from nobodies posting on support threads, but from large corporate customers, the ones Microsoft listens to carefully. This problem can't be allowed to continue.

Behind the Curtain of Second Tuesdays: Challenges in Software Security Response (from Microsoft's Channel 9)