VMware bug causes worldwide disruption

VMware bug causes worldwide disruption

Summary: VMware virtual machines on all hosts with the company's latest hypervisor, ESX 3.5 Update 2, in enterprise configurations have found that it will not power on after being turned off.


Update at 8:35 a.m. PT on Wednesday: Since ZDNet UK published this article, a patch for the flaw has been posted to VMware's Web site.

VMware virtual machines on all hosts with the company's latest hypervisor, ESX 3.5 Update 2, in enterprise configurations have found that it will not power on after being turned off.

The hypervisor refuses to start when the date is August 12, with customers around the world discovering the problem as midnight was passed in their time zones. A flaw in the VMware licensing code is responsible, according to Martin Niemar, group manager of virtual-infrastructure product marketing at VMware.

"We had an issue with 3.5 Update 2. It's actually a licensing problem," Niemar said. "Currently, what we know is that licensing prevents new virtual machines from powering up after shutdowns, and it prevents virtual motioning--moving a virtual machine from one host to another."

Niemar said VMware does not have a patch but that working on one is a "top priority."

"Customers should not stop virtual machines. Keep virtual machines going until we release a patch," Niemar had said. "You can also move the clock backwards on the server."

Some organizations cannot turn server clocks back for legal or technical reasons. Niemar said that, if customers have to turn machines off, and cannot turn clocks back, there is currently no fix. (Editors' note: A patch is now available on VMware's site.)

Niemar could not commit to a time line for a patch, nor could he comment on forum claims that the fix will first be available to customers as a complete reinstallation from ISO or TAR images, with a patch for installed code coming later.

"We understand the bug," he said.

VMware first learned of the issue when Asia-Pacific customers started to come online on August 12. Technical issues have been discussed on the VMware communities blog.

"VMware engineering has isolated the root cause of this issue and will reissue the various upgrade media, including the ESX 3.5 Update 2 ISO, ESXi 3.5 Update 2 ISO, ESX 3.5 Update 2 upgrade TAR and ZIP files by noon, PST, on 13 August," one poster wrote. "These will be available from the page: http://www.vmware.com/download/vi. Until then, VMware advises against upgrading to ESX/ESXi 3.5 Update 2. The Update patch bundles will be released separately, later in the week."

At the time of writing, ZDNet UK was unable to confirm this comment.

Topics: Security, Hardware, Virtualization, VMware

Tom Espiner

About Tom Espiner

Tom is a technology reporter for ZDNet.com. He covers the security beat, writing about everything from hacking and cybercrime to threats and mitigation. He also focuses on open source and emerging technologies, all the while trying to cut through greenwash.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • "Move the clock backwards on the server"?!!

    "Move the clock backwards on the server"? What kind of critical resource management is that? No processes, no interprocess, intermachine, interdepartmental, intersite dependencies--hmm, to start, NTP...--will suffer heartburn from that and/or from its later rectification?

    "Move the clock backwards on the server"? What kind of trustworthy license management system wouldn't notice that?
  • Tom posted this OVER 12 HOURS after it was fixed!!!!

    Just saying. If you're going to claim that the patch had been released after you posted the article, you should really make sure that is actually the case.
    • Are you saying...

      ... that this report that came out at 6 AM EDT was talking about a flaw that had not happened yet?
      Confused by religion
      • Fixed before this article war written

        ~9AM on 8/12: Bug was reported to VMware and several other sites.
        ~Noon on 8/12: VMware had the workaround (set date back on ESX hosts)
        ~10PM on 8/12: Fix was released

        This article was posted on 8/13. Get your facts straight before saying that a patch hasn't been released. While I'm dissapointed VMware let this slip through their QA, I'm very impressed with how quickly they responded to the issue.
  • RE: VMware bug causes worldwide disruption

    Wow, VMWare ESX rendered useless by a licensing bug? Wow. I was actually considering VMWare for our new datacenter. I have used their freeware stuff for some time. But this definitely sells me on a GPL licensed hypervisor. I have also been using Virtualbox for some time and it works quite well for my needs. Even if it has less features here or there at least I won't have to worry about some proprietary licensing bug wiping out our company in the middle of the night.