Microsoft admits WGA failures "coming up more commonly now"

Scrolling through the posts on Microsoft's official WGA Validation Problems forum is like reading accident reports from a multiple-car pileup on Interstate 5. Many of the victims are completely innocent and have no idea what hit them, and cleaning up the mess can be a nightmare.

Even a casual reading of the posts at the WGA Validation Problems forum makes it clear that WGA has serious problems. But Microsoft refuses to share any hard data about WGA installations, making it impossible for independent observers to quantify the extent of the problems. Until now, that is.

With the help of a researcher, I went through a sample of 137 recent problem reports from actual Windows users, posted publicly on the WGA Validation Problems forum. Our research was the online equivalent of listening in to two weeks worth of calls to Microsoft's support lines. The results we found directly contradict Microsoft's insistence that "only a handful of actual false positives have been seen."

According to our analysis, 42% of the people who experienced problems with WGA and reported those problems to Microsoft's public forums during that period were actually running Genuine Microsoft Windows. That's not just our opinion, either. Those statistics were reported by the Redmond-approved Microsoft Genuine Advantage Diagnostic utility.

In our research, we discovered that two Microsoft employees have publicly and repeatedly acknowledged that a particular type of WGA false positive is "coming up more commonly now." We found a widely used security tool from McAfee that triggered WGA failures on perfectly legitimate systems. And we read dozens of reports from frustrated Windows users whose systems are running legally licensed copies of Windows XP but who are blocked from receiving security updates via Windows Update and who are blocked from installing premium Microsoft downloads such as Internet Explorer 7 because the WGA tool mistakenly identified their Windows installations as counterfeit.

Here are the gory details:

Among Windows users who submitted WGA problem
reports to Microsoft's public forum in a two-week period
in August, 42% were running Genuine software, as
confirmed by Microsoft's official diagnostic utility.

Our methodology was as follows:

• We reviewed all discussion threads from the WGA Validation Problems forum, beginning with threads started on August 1 and continuing in sequence until we reached new discussions dated August 15. Choosing this range of dates allowed us to be certain that Microsoft representatives had had sufficient time to respond to every post. We also looked at a sample of more recent posts and found reports that were similar to those during the sample period.
• We counted only forum threads containing output generated from the Microsoft Genuine Advantage Diagnostic utility. Microsoft's representatives insist that users run this utility and paste the results for analysis before they will agree to resolve any issues on this forum. This effectively eliminated "chatter" and posts that didn't directly relate to WGA.
• We tabulated the Validation Status field to divide the total sample of problem reports into the "buckets" Microsoft uses to classify Windows users for its WGA program. The overwhelming majority - all but 6% - of the validation results fell into four categories: Genuine, Blocked VLK, Invalid Product Key, and Not Activated.

As the graph shows, 39% of problem reports were from people who were indeed using counterfeit software, activated by an invalid product key or a stolen or leaked volume license key that has been blocked by Microsoft. But we were shocked to discover that the largest group of reported problems - representing 42% of the reports in our sample - came from people running copies of Windows that were Genuine, according to the MGA Diagnostic tool.

We have every reason to believe that this group is a representative sample of people who have experienced unexplained WGA notifications telling them they're running counterfeit software. (Obviously, it doesn't include people who knowingly installed counterfeit copies of Windows.) If anything, they represent a slightly more sophisticated group than average, because they were able to track down the WGA Validation Problems forum. But there's no indication that this group is otherwise atypical.

So, where did those false positives come from?

One large group consists of people who, for some unexplained reason, were displaying cryptographic errors related to digital signatures. The problem is so common, in fact, that Microsoft representatives have a canned response they paste into replies to forum visitors who appear to be showing false positives caused by these errors. Here's a sample of the canned text, posted by Microsoft's Phil Liu. We read these exact same words over and over and over again in forum threads during our sample period:

The issue seems to lie with the "unknown" signature that is coming up more commonly now. The "unknown" signature denotes a problem with detecting digital signatures. [emphasis added]

That snippet - "unknown signature ... coming up more commonly now" - appears in at least 30 different threads between July 31 and September 18. The solution isn't easy, especially for a computer novice. Microsoft's representatives instructed users to open a Command Prompt window and type 10 separate commands to re-register system DLLs. The repair procedure worked, but this victim's response was typical:

That fixed the problem. I was able to get the updates and no more counterfeit messages. I think there is an issue with this new validating software. I am for stopping piracy - but this is crazy.

Another set of problems were caused by a registry-cleaning utility called QuickClean, which is part of McAfee's Internet Security Suite. According to McAfees' promotional copy, "McAfee QuickClean technology helps optimize your computer performance, eliminating drive-clogging 'Internet build-up' (e.g., temp files, cached files, file remnants, Active X code), unused programs and other unnecessary clutter to free up valuable disk space." Unfortunately, it also "cleaned up" the information the WGA utility used to identify legitimate copies of Windows XP.

A post on McAfee's support forums first reported this problem on July 31. This thread, started on August 11, is the first to document the problem on Microsoft's support forums:

One of the tools on the new Security Center is a "quick-clean" tool, which I ran because my computer was running a bit slow. The next morning, after a McAfee security (definitions) update and a reboot, WGA flagged my computer as non-genuine.

Over the next three weeks, another nine users added posts to this thread saying they were experiencing identical problems. Microsoft's Phil Liu posted an update on August 31, confirming that McAfee had finally issued a patch on August 30. In other words, users of a very popular security suite for one full month were one click away from falsely being accused of running counterfeit software. That problem is now solved, but there's no indication that WGA is robust enough to protect itself from other system-level utilities that might cause similar problems in the future.

And then there are the Microsoft customers who receive no help at all after reporting that WGA notification messages were flagging their software as counterfeit even when the MGA Diagnostic utility showed it was Genuine. Most get canned responses telling them to go visit Microsoft's WGA Diagnostic page or update the WGA Notification utility or run a command to re-register the Wgatray.exe program. This thread is typical, with two separate customers reporting that the canned responses didn't work and no follow-up from Microsoft. We found dozens of these cut-and-paste responses to Microsoft customers reporting that their Genuine software had failed WGA validation. Did the fixes work? No one knows, because the original posters either never returned to the forum or never posted a reply. Only 20% of the forum threads we looked at included a follow-up message from the original poster indicating that they had solved the problem.

And the reports we analyzed here are from customers who actually managed to find their way to the WGA Validation Problems forum. On our test machine, running a counterfeit copy of Windows XP supplied to us by Microsoft, clicking the pop-up WGA Notification bubble led to a page that offered to sell us a Windows Genuine Advantage Kit for $149. The page includes no acknowledgment that the errors might be caused by problems with digital signatures, with third-party software, or with a failed WGA Notification installation. Since I published Busted! What happens when WGA attacks (including this Image gallery showing the WGA process at work), Microsoft has made no attempt to improve the help it offers users who may be experiencing false positives.

How many legitimate customers are simply paying Microsoft an extra $149 because it's easier than going through the hassle of working out the problem? If the answer is more than zero, it's too many.

Last Thursday, I contacted Microsoft's WGA team and offered to discuss the details of this story with them so they could comment on it. Despite repeated follow-up messages from me, they have declined the opportunity to hear about this story or to comment on it.

Update 26-Sep 6:15AM PDT: After this story was posted, a Microsoft spokesperson who had not read the story and had declined the opportunity to review any details about our findings sent an e-mail statement affirming the company's confidence in WGA. You can read that statement in this follow-up post. 

Want more background on WGA? Read my previous reports.