Defective McAfee update causes worldwide meltdown of XP PCs

Defective McAfee update causes worldwide meltdown of XP PCs

Summary: Oops, they did it again. Early this morning, McAfee released an update to its antivirus definitions for corporate customers that mistakenly deleted a crucial Windows XP file, sending systems into a reboot loop and requiring tedious manual repairs. It's not the first strike for the company, either. I've got details.

SHARE:

[Update, April 22. More details in my follow-up post, McAfee admits "inadequate" quality control caused PC meltdown.]

Oops, they did it again.

At 6AM today, McAfee released an update to its antivirus definitions for corporate customers that had a slight problem. And by "slight problem," I mean the kind that renders a PC useless until tech support shows up to repair the damage manually. As I commented on Twitter earlier today, I'm not sure any virus writer has ever developed a piece of malware that shut down as many machines as quickly as McAfee did today.

Here's how the SANS Internet Storm Center describes the screw-up:

McAfee's "DAT" file version 5958 is causing widespread problems with Windows XP SP3. The affected systems will enter a reboot loop and [lose] all network access. We have individual reports of other versions of Windows being affected as well. However, only particular configurations of these versions appear affected. The bad DAT file may infect individual workstations as well as workstations connected to a domain. The use of "ePolicyOrchestrator", which is used to update virus definitions across a network, appears to have [led] to a faster spread of the bad DAT file. The ePolicyOrchestrator is used to update "DAT" files throughout enterprises. It can not be used to undo this bad signature because affected system will lose network connectivity.

The problem is a false positive which identifies a regular Windows binary, "svchost.exe", as "W32/Wecorl.a", a virus.

McAfee now has its own KnowledgeBase page posted, with details about the problem and the fix. The symptoms are described, tersely, as "Blue screen or DCOM error, followed by shutdown messages after updating to the 5958 DAT on April 21, 2010."

Update: Engadget's Nilay Patel quotes a statement from McAfee downplaying the impact on consumers:

The faulty update has been removed from McAfee download servers for corporate users, preventing any further impact on those customers. We are not aware of significant impact on consumer customers and believe we have effectively limited such occurrence.

That's bad news for McAfee. Corporate customers are likely to tally up the one-day cost of fixing this damage (or multiple days, if Engadget's report of tens of thousands of affected PCs within single companies is accurate), and they're likely conclude that it's time to find a new supplier of security software. At the very least, McAfee is going to have a lot of explaining to do at contract renewal time.

McAfee says it has already replaced the faulty virus definitions with an updated set, so if you update your definitions using the most recent set you will not encounter this issue. The company's official recommendation for repairing the damage involves copying Svchost.exe from a working system and manually copying it to an affected system. The McAfee technical bulletin doesn't include details about how to get to a command prompt on a system that's been temporarily bricked. (Using an XP installation disk allows a tech support professional to boot to a recovery environment and copy the necessary files from a command prompt. The good folks at BleepingComputer.com have published a tutorial that explains the process. Third party recovery tools also provide access to the file system from command-line environments.) This sort of repair is not a job for end users, certainly, and generally requires a skilled support professional.

Update 2: An e-mail correspondent from a large U.S. company  (see full text at end of this post) says that multiple files in addition to Svchost.exe mght be affected and claims that simply replacing Svchost.exe might not be enough to repair the damage. I'm still looking to confirm this report.

Update 3, 22-Apr: McAfee has released a repair tool it calls the SuperDAT Remediation Tool. Details are on this page. Running this tool is still a manual process that requires booting from portable media and running the executable file, in safe mode if necessary. 

Now, it is hard to imagine picking a more crucial file to torpedo. Svchost.exe is one of the most crucial of all Windows system files. It hosts the services that make just about every OS function possible. As the symptoms described here suggest, Windows simply won't start if Svchost.exe isn't there.

The bigger question is how on earth an update like this ever made it out of the testing lab and onto a production server. This should have been caught at the very beginning of the testing process.

Unfortunately, though, this isn't the first time McAfee has had a screw-up like this. Back in 2009, when the Conficker worm was making the rounds, I took a close look at how McAfee was handling its response to the new threat and was appalled at the sloppy, error-ridden documents they published for consumers and IT professionals. Here's what I wrote at the time:

Security is serious business, and details matter. When a company as large as McAfee is this sloppy with its public response to a high-profile issue, it makes you wonder how tightly the engineering, development, and support sides of the business are being operated.

Now we know.

Ironically, one company that was apparently affected by this issue is Intel, which was identified by the New York Times. It's the second major security headache for Intel in six months, following a widely publicized breach of its systems in China around New Year's. (Intel acknowledged the "recent and sophisticated incident [that] occurred in January 2010" in its 10-K report filed with the SEC earlier this year.)

If you've been affected by this issue, leave a comment in the Talkback section, I'll add further details as I come across them.

Update: I'm beginning to hear directly from people who were affected by this coloassal screw-up. One correspondent says he just fixed over 300 PCs: "Looked so much like Blaster from way back. Horrible clean up too as no network access. Moving clients to something with more centralized control ASAP."

A report from a university IT pro says 1200 PCs on his network were knocked out.

Another e-mail from an IT pro at a large U.S. company says that "hundreds of users" in his organization were impacted:

This issue affected a large number of users and is not resolved by simply replacing svchost.exe.  You must boot to safe mode, then installl the extra.dat, then manually run the vscan console.  You then remove the quarantined files.  All users had at least two and some had up to 15.  Unfortunately, using this method, you have no way to determine if some of the files you are restoring are vital system files or virus files.

I'm still hoping to get confirmation from Intel, where at least one anonymous source says "tens of thousands of PCs" were hit.

A report from Australia says 10% of the cash registers at the country's largest supermarket chain were knocked out, forcing the closure of 14-18 stores.

Via e-mail, I've heard firsthand reports from people who had to manually repair PCs at some very large corporations and arms of the U.S. military.

Topics: PCs, Hardware, China, Software, Security, Operating Systems, Networking, Malware, Intel, Windows

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

389 comments
Log in or register to join the discussion
    • Apple?

      There is no McAfee for Apple, and none is needed. There is no
      svchost.exe for Apple, and none is needed. Oh, your just one of those
      Linux-zealots who spews Linux, Linux, Linux everywhere hoping that
      people won't notice the lack of key industry standard applications, the
      requirement for geek-knowledge to make anything work and that the
      free apps may be substitutes, but sure aren't replacements for the
      commercial apps they are using now.

      Sorry, my mistake.
      Asiafish
      • No McAfee for Apple?

        "There is no McAfee for Apple, and none is needed. "

        That's interesting, because both McAfee and Symantec got their start with an anti-virus program for the Apple Macintosh.
        cdgoldin
        • Classic vs OS X

          You're talking about Mac OS "classic", rather than Mac OS X. They are as
          different as two operating systems can be (despite the similar name).
          Jeremy-UK
        • Not McAfee, SAM was...

          Back in the days the only commercial Antivirus for Macs was SAM (Symantec Antivirus for Macintosh) and the last one was shipped in 1994-1995, there was one Norton Antivirus for Mac after that, it didn't sell well, so they discontinued it.

          McAfee has just been recently available for Macs advertised as "Complete protection for PowerPC and Intel-based systems". Which means it is for intel based Macs. The freeware antiviruses that existed Disinfectant and ClamXav are (I think) gone for good.

          I'm not saying we'll not need them one day, but for now we're ok.
          Liliana Pubill
          • RE: Not McAfee, SAM was

            Sorry, but I stock Norton Anti-Virus for Mac in my store.
            Ghostman52
      • McAfee for Mac

        Yes, there is a Santa Claus. And there is a McAfee for Mac.
        McAfee Security for Mac for OS X 10.4.11 - 10.6.x
        McAfee Virex 7.7 for OS X 10.3.x
        McAfee VirusScan for Mac 8.5 for OS X 10.4.x

        Not sure how useful they are, but since my company has a site license, I
        have McAfee Security for Mac installed.
        levinson
        • McAfee for linux too.

          VirusScan Enterprise for Linux

          I didn't want anyone feeling left out.
          Alzie
        • ??

          What is it allegedly protecting you from?
          rag@...
        • I wouldn't install McAfee on any computer

          unless it was my sworn enemy's only computer. I learned McAfee was purely crap about a decade ago and would'nt wish it on any one, especially where it will never be needed like on my Mac.
          jacarter3
      • No McAfee for Apple?

        "There is no McAfee for Apple"? Really? Someone better tell McAfee:
        http://www.mcafee.com/us/enterprise/products/system_security/clients/virusscan_for_mac.html
        kwagner_z
      • Apple?

        You are just so wrong on so many fronts. Talk to me at richdave2000@yahoo.com and I will be more than happy to tell you why.
        richdave
      • Why does everything

        degrade so rapidly to Apple-vs-the world?
        Papa_Bill
      • Apple.

        He doesn't sound like a zealot, you sound like a
        flat-Earther.
        apostate
      • RE:Apple?

        "the
        requirement for geek-knowledge to make anything work and that the
        free apps may be substitutes, but sure aren't replacements for the
        commercial apps they are using now."
        Really?? This is your argument?
        Dude you are a dork.
        I have been into computers SINCE the early 80's. And this is your argument?!? In lots of cases the replacements are comparable to commercial Apps. OH, BTW, that is how I have made mine through out the years is by being "the geek" and my "geek knowledge base". Dude you are an amateur!
        enawn
        • In lots of cases

          The Free Software is Commercial Software.

          What is being referred to here as Commercial software is in reality Proprietary Software.

          Free (as in Free Open Source Software) Software can also be Commercial Software (in fact the GNU GPL expressly states that software licensed under that license may be sold "for whatever price the market will bare", What does anyone think Red Hat's Distribution of Linux (mostly licensed under the GPL) is if it's not Commercial Software.
          tracy anne
        • Geeks my @ss

          Hey,
          Nobody gives a rats ass about what you're using on your personal machine down in the basement or garage.
          Ever tried to port 150 hospital clinical apps written for Windows to Linux or whatever?
          You can start now. And then support them. To the end of your days. We have 12000 machines waiting for your "geek knowledge base" to be applied.
          maxtov@...
          • You are behind the eight ball, NOT on the cutting edge

            Ever hear tell of VistA?

            http://www.worldvista.org/AboutVistA/copy2_of_index_html
            Open Source VistA Platform

            The complete Open Source VistA stack consists of:

            GNU/Linux
            The combination of the GNU tools and the Linux kernel, which together provide a robust, scalable OSFS operating system.

            VistA
            Veterans Health Information Systems and Technology Architecture, a Healthcare Information System (HIS). VistA is widely believed to be the largest integrated HIS in the world. It was originally developed and maintained by the U.S. Department of Veterans Affairs, based on the systems software architecture and implementation methodology developed by the U.S. Public Health Service jointly with the National Bureau of Standards. It is designed to provide a high-quality medical care environment for the country's military veterans. VistA has a proven track record of supporting a large variety of clinical settings and medical delivery systems. VistA is in production today at hundreds of healthcare facilities across the country from small outpatient clinics to large medical centers. The software is currently used by the Indian Health Service and a number of other healthcare organizations around the world.

            http://en.wikipedia.org/wiki/VistA
            The Veterans Health Information Systems and Technology Architecture (VistA) is an enterprise-wide information system built around an electronic health record, used throughout the United States Department of Veterans Affairs (VA) medical system, known as the Veterans Health Administration (VHA).[1]

            By 2003, the VHA was the largest single medical system in the United States,[2] providing care to over 4 million veterans, employing 180,000 medical personnel and operating 163 hospitals, over 800 clinics, and 135 nursing homes.
            Ole Man
      • Apple has their problems...

        http://secunia.com/advisories/search/?search=apple

        Linux, Mac, and PC.. meh. I like what ever works. Why fight over an OS. Linux is cool because it's free and you can do just about anything with it. Macs are cool because they are pretty, but they cost too much (some people like that). Windows good because it is what we know (it took me a while to even come up with that one).

        I'd rather debate iPod or the MS Zune player!

        P.S. I pick the Zune. :)
        Senrats
      • Only obscurity saves apple

        Considering that OS X is considerably less secure than windows 7/vista (based on the results of the hacking contests), and considerably slower to react to known vulnerabilities than MS, it would seem that the only thing saving apple from the same virus issues is its small global marketshare. It also has virtually zero presence in the locations where a lot of hacking hits come from, like China and Russia so many hackers simply don't have experience with the platform.

        If apple continues to grow it will, sooner or later, become a victim of its own success and enter the same virus rat race windows has to deal with.

        Linux OTOH, is both obscure and secure, whatever its other faults may be.
        SlithyTove
        • If an Apple had a worm

          Would anyone care?

          Not really
          kandrew@...