Hypervisors: The cloud's potential security Achilles heel

Hypervisors: The cloud's potential security Achilles heel

Summary: A cloud is only as secure as the hypervisors that support its virtual machines and how secure are those? That's a darn good question and one we tend to avoid looking at.


Napa Valley, CA: When some security companies talk about potential threats they tend to, it hypes the danger. When someone who works with cloud technology, like Linux kernel developer and cloud company Nebula's senior security software engineer Matthew Garret talks about potential security problems in cloud computing, I sit up and take notice.

Matthew Garrett
Matthew Garret, Linux hacker and top cloud security expert, fears we're not paying enough attention to potential zero day attacks on hypervisors.

Best known for finding ways to get Linux to work with Windows 8's secure boot, Garret is also both a low-level security and cloud expert. At the Linux Foundation's Linux Collabration Summit, Garrett explained that his greatest worry is hypervisors.

Hypervisors, as ZDNet's own Dan Kusnetzky explained in his book, Virtualization: A Manager's Guide, "can run one or more complete virtual systems on a physical machine. Each of these systems—such as Linux's KVM, Microsoft's Hyper-V or  VMware's vSphere/ESXi—process as if it has total control of its own system, even though it may only be using a portion of the capabilities of a larger physical system."  

It is the virtual machines (VMs) that the hypervisors automatically generate on servers on demand that we're using when we work on the cloud. Without modern hypervisors, the cloud simply couldn't exist.

So, the real security question for the cloud starts with: "Can you trust your cloud provider's hypervisor?" Garrett's answer is "maybe."

Garrett said, "On the balance of probabilities, you have to assume that hypervisors probably do contain vulnerabilities, that they do contain flaws that can be exploited to gain access and allow guests to break out into the hypervisor."

Has anyone found such a bug yet? No. Is the potential there for such a security hole? Garrett says yes.  

Can hypervisors detect such attacks? Uh… no, not really.

Say for example, "You host with Amazon. You have no idea what else is running on the same hardware, you have no way of seeing the other guests, what services they are running. It's conceivable that your main data store and credit-card processing system VMs are on the same hypervisor as someone running a bad PHP-based personal Web server VM. Say that amateur's VM is hacked. The hypervisor should protect other guest machines from a compromised guest, But, what if the hacker can then break into into the hypervisor. Then your otherwise protected, up-to-date secure VM guest can potentially be attacked from an unexpected direction."

"Is it absolutely certain that if someone compromises a guest on the same hardware as you, that that compromised guest will then not be able to break into the hypervisor, and then from the hypervisor compromise your system?" Garrett asked. The answer's no.

He continued, "Humans are bad at writing code. You should tell your kids at night that hypervisors are secure, but while we've done well, none of them are perfectly secure. There will be bugs."

"Once someone gets to the hypervisor then it's game over, everyone can be compromised," said Garrett. "None of this will be your fault, but you'll have been violated. You can have a perfectly secure VM, but if the hypervisor isn't secure, than your security doesn''t matter.

Therefore, Garrett said that you must ask your cloud providers hard questions. These include how are VM guests isolated? If a hypervisor security issue is found, how does the provider respond? What mechanisms are used to detect compromises? Can a cloud provider say, with certainty, that a host machine has been compromised in a fundamental way? And, if so, what tools will  they use to investigate?"

These are very difficult questions," Garrett continued, "and cloud providers are extremely reluctant to answer them." The reason for this is simple, if you doubt the security of hypervisors, you doubt the very foundation of cloud computing. For instance, "The entire public statement from Amazon about guest security is that 'the hypervisor protects guests from interfering with each other.'"

Garrett's pretty much right about Amazon's cloud hypervisor security position. More troubling still, Amazon Web Services' (AWS) security statement hasn't been updated since December 5, 2008. Things have changed a wee bit since then when it comes to clouds, hypervisors, and security.

Garrett admits that "So far there has been no proof that this has ever happened, but we must assume that people are looking for hypervisor zero days. Hypervisor security is difficult. They have large attack surfaces with lots of entry points, their code is complicated, and they can be dependant on subtle CPU behavior."

Worse still, he continued, "We don't have best practices for checking on hypervisors for security violations. We're beginning a new security journey and we need to move faster. In one worst-case scenario, your entire cloud and all the VMs on it will be as vulnerable as the weakest guest."  

Related Stories:

Topics: Virtualization, Cloud, Open Source

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • Ouch!

    Potentially more devastating than most security problems because the real compromised system is not the one hacked.
    • the expectation is that at least one VM was hacked...

      And through that one the hypervisor is hacked.

      I would think that the most vulnerable spot would be the video emulator...

      With a second weak spot going though the network to come back to the host - as now it it would be a local connection.
  • It could, actually be worse

    Quoted from Matthew Garrett in the article:
    "t's conceivable that your main data store and credit-card processing system VMs are on the same hypervisor as someone running a bad PHP-based personal Web server VM. Say that amateur's VM is hacked."

    Malware miscreants, under the guise of a front company, could have their own VM on the same hypervisor. The miscreants have already moved to the Cloud in support of many of their misdeeds. In this case, no VM hack is necessary and there will not be even be a possibility for an amateur or anyone else discovering that their own VM has been hacked, causing them to alert Amazon (as an example).

    Given the current state of the Internet, it would be prudent to *assume* that the malware miscreants control a VM on a "public" hypervisor. Those that can afford it might want to consider paying for an entire hypervisor where they control the security, including configuration, maintenance and monitoring, of the VMs.
    Rabid Howler Monkey
    • I agree

      It's happened before and it will happen again:

      Alan Smithie
      • Forgot to add

        Especially as the NSA has backdoors (possibly) in CPU microcode
        Alan Smithie
  • Refreshing article, Steven

    Why, refreshing? Because neither Matthew Garrett nor Steven, in the article, trotted out the "many eyes" tired trope for open source software, including the KVM and Xen hypervisors..

    Quoted from Matthew Garrett in the article:
    "They [hypervisors] have large attack surfaces with lots of entry points, their code is complicated, and they can be dependant on subtle CPU behavior."

    The Qubes OS website (note that Qubes OS uses the Xen hypervisor for desktop computing) speaks of the relatively small Xen code base and makes the assertion that it translates to a smaller attack surface:

    "Why Qubes uses Xen, and not e.g. KVM?
    In short: we believe the Xen architecture allows to create more secure systems, i.e. with much smaller TCB, which translates to smaller attack surface."

    I'd like to read someone's (esp. Matthew Garrett) take regarding this assertion.
    Rabid Howler Monkey
    • Xen/KVM security

      I don't absolutely buy it - the number of security issues in Xen isn't massively lower than KVM. One significant benefit KVM has is that if someone is able to escalate as far as qemu but not as far as the kernel, you can use existing Linux confinement mechanisms (such as apparmor or selinux) to prevent cross-VM attacks. But yes, Xen's smaller attack surface ought to be advantageous here.

      (But then our product is KVM-based, just as Qubes is Xen-based, so obviously take all of this with a pinch of salt)
      • A bit of a late reply, but ...

        Just like KVM can use Linux Security modules such as apparmor, selinux, etc. to prevent cross-VM attacks and attacks on the hyper-visor, Xen can use Xen Security Modules (or XSM) implemented via Flux Advanced Security Kernel (or FLASK) to do the same. More here:


        P.S. Sincerely appreciated your response to my comment. :)
        Rabid Howler Monkey
  • The solution...

    ...of course, is to run each customer's VM's under credentials which only have access to their own VM's. Even if someone breaks through to the hypervisor host, they would be contained by the security on the host itself and prevented from monkeying with other people's VM's.

    Hypervisors will always have flaws, just like any other software. The solution as always, is defense in depth.

    The question is: How many cloud providers will actually do the right thing here?
  • What about the liability issue?

    Everyone else is discussing the hypervisor, but I will ask regarding the legal angle. What are Amazon's terms and conditions with respect to breaches? If your VM is Targeted (!), is Amazon on the hook for the damages or are you just SOL? If the latter, then Amazon is really just selling a black box with a big "trust me" sign written on the side. I would never use a cloud service because of the risk, but if I did I would demand to understand their security before I transferred any data or operations to them.

    P.S. I would expect Jeff "social Darwinist" Bezos' cloud service to include no insurance or liability assumption whatsoever. Target's liability is in the billions of dollars; what will yours be?
  • Interesting topic...but...

    Im in no way an expert in hypervisors in any way, but unless Im mistaken, it had always sounded like this was the least likely way anyone would attack any system due to its great difficulty and high potential for lack of success combined with some lack of certainty of what could be gained in such an attack, particularly given its over all difficulty?

    Isnt it the case that there would be other significantly easier attack vectors that would be of more interest to just about all hackers, perhaps even with a better idea of what the payoff might be?

    True or false...isn't this just one more case of what is physically possible as opposed to what has some reasonable possibility of success or even have some significant interest in a hackers mind?

    While Im not one for having much interest, or sympathy for the "cloud", this is the first Ive heard of someone talking about some threat on hypervisors being the Achilles heel to the whole thing.

    As far as Im concerned, the Achilles heel to the cloud is the effect of turning over control of so much of your computing and internet interests to some company willing to rent you services, storage, applications and all forms of entertainment from the web. And of course, on top of those fees is still going to be your monthly connection fee from your ISP.

    Essentially; what started out as looking like you simply purchase your device (computer) pay a monthly flat fee to get hooked to the net and hey away you go, access to a bazillion webpages, and if you want special games or applications you lay out a one time purchase fee and that's it for whatever game you want or application you want, going from that, to going to purchase your hardware, pay your monthly connection fee, rent your games by the month, rent your applications by the month, pay some cloud service a monthly fee to store your data, (you know, the multitude of vacation pictures, videos and documents you used to store on your 1 or 2 TB HD for next to nothing), pay for your entertainment pay for your just about anything.

    That's what the clouds Achilles heel is.

    What happened to the concept that a computer is a computer and the internet is the internet and the two are different things even though you use a computer to connect to the internet?

    How did we get from the computer is a computer dictum to the internet can be your computer vision? Why should we think that's an advancement? Exactly?

    Surely not simply because we could then get on any dumb terminal and connect to the resources in the cloud that now represent "your computer"??

    I can clearly see that monthly cloud rental fees for the big IT companies, who are sick at the thought of the software industry going the way of Windows XP, where you purchase something once and then you are good for 10 years. Or more.

    I can clearly see that just about everything you get or put onto your computer having to come from some "trusted and paid for" source in the cloud is a great way for the big IT companies to not only improve their revenue stream but to eliminate current modes of most software piracy. After all, if you have no DVD drive, no USB, and your net connection is the only way to get something new on your computer, and your computer will not load or install anything not coming from a "trusted source" than it sure solves a lot of piracy.

    This of course all sells well under the rubric of improved security. I guess that sells well for those who honestly feel they have suffered in some way through lax security over the last couple of years. Hmmmm. It seems to me the last nasty security incident I suffered was about midway through 2005. I certainly cannot buy into the cloud because it helps my hurting security. My security hasn't been hurting in a long long time.

    I certainly cant buy into the cloud because internal storage cost me so much. Anyone around here look at the price of a good 2TB hard drive lately??

    I cannot buy into it because of promises from the big IT companies telling me how nice it will be for me to have ready access to the most current software without paying full freight for new updated releases as soon as they come out. Like most people, I don't need new updated releases every year, or whenever they decide to release new software. I cant imagine the big IT companies want to have us move to the cloud because it will cost us less and make them less money by charging monthly or yearly fees.

    I cannot buy into it knowing that if I am relying on the cloud for access to much of my own data and for various applications and bits of entertainment and gaming I normally would have had stored on my own HD's I would now find myself relying on my internet connection to get at ANY of that I have in the cloud.

    But...I kind of have doubts that hypervisor security is going to become a drawback to using the cloud. In any event, it sounds like a third rate concern about a new way of doing things that already has more than enough drawbacks to worry about.