Bare metal hypervisors: a group test

Introduction
At its core, there are two major types of virtualisation: Type 1 and Type 2. There are complaints attached to these labels, and there are hybrids that defy them, but by and large they serve their purpose in helping us to categorise virtualisation.

Type 1 is known as a native, or 'bare metal' hypervisor. It requires no operating system to be preinstalled — it is one, just in a very stripped-down form. If your intent is virtualisation alone, this is the most efficient type, allowing access to the hardware through the hypervisor.

Type 2 on the other hand, is run on a preinstalled operating system. It's also known as hosted virtualisation or a virtual machine monitor (VMM). You'd likely run this when you need the host OS's features to have access to your hardware. For example, you may want to run on Solaris to give direct access to creating a RAID-Z array. Or perhaps you need to run on Windows, to give a virtualised operating system access to a hardware device that usually only Windows can see. Or maybe you just want to test out a new OS before doing a roll-out — either way, different virtualisation suits different needs.

In this round-up we'll be looking at Type 1 virtualisation solutions from the market leaders: Microsoft, Citrix and VMware. In the coming weeks we'll look into Type 2 competitors Parallels, VirtualBox, VMware Workstation and a few odd tools that aren't quite virtualisation, but do get binaries running across different OSs.

Bare metal hypervisors
In today's market, VMware still retains dominant share, but others are starting to move in on its lead, including XenServer from Citrix and Hyper-V from Microsoft. Both have received a large amount of press and also support from the industry. Of course, there are other Xen-based solutions such as xVM Server from Sun and Red Hat Linux Enterprise (RHEL) Server, which currently include virtualisation (although RHEL will be moving to KVM later in the year, as Ubuntu did last year). For this review, our focus was primarily on VMware ESXi 3.5, XenServer 5 and Hyper-V R2 beta (more on why later).

How we tested

Our objective was to perform a set of (repeatable) tasks on virtual machines running on top of the various hypervisors and measure the performance of these tasks.

The server we used for this testing was kindly provided by Xenon. Featuring dual quad-core Intel Xeon E5462 processors, 8GB of RAM and a 750GB hard drive, it runs on a Super Micro X7DWU motherboard with ATI ES1000 graphics and an Adaptec 3405 RAID controller.

Xenon's physical server was first tested with a native install of Windows Server 2003 SP1 to establish a performance baseline using the eight CPU cores and 8GB of RAM available. The hypervisors were configured with four virtual CPUs and 3.5GB of RAM (leaving 1GB free for the hypervisor). This is the maximum configuration possible for all the servers, except for XenServer, which supports up to eight virtual CPUs. A fresh install of Windows Server 2003 SP1 was then installed on the configured virtual machine (VM) on each of the hypervisors in turn, and the following tests were run:

• The OS was upgraded from SP1 to SP2, the time taken was recorded
• Sungard was run and the result recorded
• Cinebench single CPU was run and the result recorded
• Cinebench xCPU was run and the result recorded
• A 600MB file was copied up and down from a network share
• A 4GB file was copied up and down from the network share

VMware Server 2, a Type 2 virtualisation product, was then tested as another control to see how hypervisors performed.

Testing procedure
The testing procedure was divided up into the following categories:

• Product features and function — maximum configuration sizes
• Product accessibility — how easy it is to download and install
• Manageability — how easy it is to create and manage virtual machines

Product features and functions
The following table summarises each of the products' configuration limits:

Product accessibility

Each product was assessed based on how easy it is to obtain and install.

Citrix XenServer
Downloading XenServer from Citrix is simple. After navigating to the XenServer product page, it's a simple matter of filling in some details before being directed to the XenServer download page. Two files are required, an INSTALL and LINUX ISO, totalling a little over a gigabyte.

Of all the hypervisor installations, XenServer was the simplest. It merely required booting the server from the install CD and following its prompts to enter details such as its IP address and server name. It should be very familiar to anyone who has installed Red Hat since the Domain 0 kernel is based on CENTOS. The install CD is then also used in a Windows machine to install the XenCenter management utility (not web-based). Within 10 minutes you're ready to start creating virtual machines.

VMware ESXi 3.5
VMware ESXi 3.5 is also simple to download from the VMware website. We created an install CD and the installation was straightforward. Screens are a little more polished than XenServer, reflecting this product's maturity. A web-based management console is installed on a windows machine from a separate download. The whole process takes a little longer than XenServer, but it's still easy for a novice to negotiate.

Microsoft Hyper-V Server 2008
Hyper-V Beta R2 was used for this test since the release version was unable to recognise the latest Intel PRO Gigabit on-board NICs in the server, reporting 'No Network on Boot'. Unlike the previous two hypervisors that use a Linux-based kernel, Hyper-V is wrapped within an appliance version of Windows Server 2008 (in fact, Windows 7 Server for the Beta R2).

The download is easy, and installation of the hypervisor onto the server is also fairly straightforward. Configuration is done via a terminal screen, which launches once you're logged in. We found this a little tedious, with lots of options to understand and make selections from.

The Windows Hyper-V management console is an MMC snap-in, which needs to be installed on either a Windows 2008 Server or a Windows Vista machine running the administration tools (x86 or x64, along with RSAT) — XP is not supported. It took us some time to understand what was required to get this working. Updates had to be applied, and the TestLab staff were unable to get the virtual machine manager running on a Vista machine, which requires a significant amount of voodoo to get working, and is nothing short of painful.

We ended up installing Windows 2008 server in order to manage the Hyper-V server. Management of the Hyper-V server is not possible until the firewall is turned off, or several services cleared (either via remote MMC snap-in once you've set up remote administration properly, or the appropriate 'netsh' command at the command line). Once we were finally able to manage the Hyper-V server, it was possible to start creating virtual machines. Windows Server 2003 SP1 requires an update to SP2 followed by another (provided) update before Integration Services can be installed (Hyper-V's equivalent to Xen and VMtools).

Manageability
Each product was assessed to determine the ease of host server management (using the console provided).

Citrix XenServer
Once again, XenServer has the most features, but it's only a Windows-based application (as opposed to web-based). A web-based console is available, but it's not supported. The XenCenter Management Client makes it easy to create, backup and even copy virtual machines (this is the only console that allowed us to copy a VM). Performance statistics and VM configuration are also easy to view and edit.

Another advantage of the XenCenter console is that it provides all the features needed if you're upgrading your licence key to a full-featured edition of XenServer. The same console will allow you to live-migrate virtual machines, create resource pools and set up high availability. There is no need to upgrade your console.

XenCenter Management Client.

VMware ESXi 3.5
VMware has a web-based console. This is also quite easy to use and navigate, although it has fewer features than XenCenter. To access its complete capability you'll need to use VMware's vCenter Server; this server is recommended if you wish to move virtual machines or automate high availability, but comes at a cost. It also requires the installation of SQL Server Express and will not install onto a Vista machine. This, of course, adds significantly to the complexity of the environment because the vCenter Server console then becomes a critical part of your infrastructure.

VMware's web-based management console.

Microsoft Hyper-V Server 2008
Hyper-V requires you to install its Server Management tools for Vista, then find the relevant update package on the Microsoft web site in order to manage the Hyper-V server. Information available is very sketchy, and the required software is hard to find, let alone understand. As a result, the TestLab staff were unable to get a Vista client working in the time available. We compromised by installing a Windows 2008 server in order to be able to manage the Hyper-V server. Eventually after downloading and installing an update, a connection was made to the Hyper-V server and the creation of virtual machines was possible. Even so, the console disconnected a number of times.

To exit the console screen, it is necessary to enter Ctrl+Alt+Left-arrow; this was extremely tedious after a server reboot or if we wanted to view in full screen. A VM with no network configured was created. To this, a network was added, but the VM did not detect this new network. To get the network attached to the VM we had to reinstall Windows — something we put down to the tested Hyper-V software being in beta.

After the ease of setting up XenServer and VMware ESXi, Hyper-V was a task. The Hyper-V console doesn't have any of the nice performance graphs that VMware and XenServer provide, although CPU usage is available. For full features you will need to use System Center Virtual Machine Manager (SCVMM).

Hyper-V management screen.

Test results
The table and graphs below display the test results: the baseline physical server is shaded in pink, while the host-based VMware Server 2 virtualisation environment is shaded a darker blue than the bare-metal hypervisors.

All times in seconds.

As you can see, the bare metal hypervisors all performed well and the results were quite close — with one exception. XenServer, unlike VMware and Hyper-V, has the ability to support up to eight vCPUs per VM, and as you can see from the Sungard test with eight vCPUs, XenServer performed exceptionally well. However, it's worth noting that its vCPUs present themselves to the Windows OS as physical CPUs, which means for eight CPUs to be selected you'll need Windows Server Enterprise Edition running on your VM.

Some interesting conclusions can be drawn from the results:

• The VMware Server 2 hosted environment only supports up to two vCPUs based on performance results. Bare metal hypervisors outperform host-based hypervisors significantly.
• XenServer, with its eight-vCPU support, performs very similarly to bare metal; the hypervisor overhead is minimal.
• In the Sungard test, XenServer with four vCPUs marginally outperforms the others, followed by Hyper-V and then VMware.
• XenServer seems to have an issue with the latest Intel NIC installed in the server, as copy-up times were unacceptably long. This was also tested on another server in-house with a different NIC (still Intel) and it worked perfectly.
• Two VMs running concurrently had minimal impact on the other VM for all the tested platforms.

Verdict
The results show that there's little difference between each of the bare-metal hypervisors. The host-based version tested is considerably slower on all tests, so choosing a bare-metal hypervisor over a host-based makes a lot of sense when hosting a server environment.

The performance difference is slightly in favour of XenServer, followed closely by Hyper-V and then VMware. However, if you wish to virtualise Citrix XenApp (presentation server), for example, then XenServer would be the bare metal hypervisor of choice.

When choosing a bare metal hypervisor you should first look at the workloads you're planning to run on the virtualised server, what your current environment is using, and what expertise will be available. If you have strong in-house Microsoft skills, Hyper-V will make the most sense. Similarly, if your IT department has Linux skills, then Red Hat or Xen will make more sense.

Forced to choose, we'd put VMware in the runner's up position, with Citrix XenServer getting our vote. It's easy to acquire, and the XenCenter Windows console provides good management with no need to upgrade when you decide to migrate to the fully-paid edition of the product.

Summary
XenServer
PROS
— Easy to install: single disc to install XenServer on the host, and XenCenter on the Windows console.
— Single console used for the free version right through to platinum editions of the product; no requirement to change or use another console when live migrating or using HA features in the full product; only console that provides a copy-VM function.
— Supports up to eight vCPUs, making it the best performer in this configuration with minimal overhead.
— Product support via forums and the XenSource community is readily available.
CONS
— XenCenter management console is a Windows application rather than a web console.
— XenServer does not have features such as memory over-commit and consolidated backup in its full products, although these are coming in future releases.
— Tool support, for example for converting physical to virtual, is not as advanced as VMware, but it's improving with each release.
VERDICT
XenServer has the most features of any free hypervisor, is the easiest to install and manage, performance is excellent and it's the only hypervisor to support up to eight vCPUs.

VMware
PROS
— ESXi 3.5 is easy to install and manage from the web console, and is the most polished of the hypervisors.
— Many advanced features in the full product highlight its maturity; best upgrade path and a strong roadmap.
— VMware is the market leader, which translates to good support via forums and the many certified engineers available in the workforce; many tools, including a capacity planner, are available to assist in the migration from physical to virtual.
CONS
— Web console is limited in terms of managing the virtual machine; requires upgrade to vCenter Server for advanced features such as Live Migration and HA.
— Upgrade from free to fully-featured version requires upgrade of console to vCenter Server, adding a greater level of complexity.
VERDICT
VMware is the market leader, which shows in the maturity of its product, the polish of its console and the vast number of support tools available — but it comes at a cost.

Hyper-V
PROS
— Best integration with Microsoft infrastructure, via MMC snap-in.
— Strong development focus from Microsoft.
— Microsoft training and development, which translates into a large number of individuals trained and certified on the product.
CONS
— Released version did not recognise the Intel Pro1000 NICs installed in the test server, whereas the R2 Beta did.
— Clunky text-based console to configure the Hyper-V server once up and running; command-line interface required to turn off firewall to enable the management console to communicate with the server.
— Integration tools did not install onto our Windows Server 2003 SP1 test server, which required us to add a legacy network connection, connect to a share to run the SP2 update, then an update before integration services would install.
— Lacking in enterprise features right now; however, an extremely aggressive and strong roadmap will see this catch up very quickly.
VERDICT
Hyper-V is not as mature as VMware or XenServer, but it has a lot of momentum and integration in a Windows environment will make this the hypervisor of choice for those running mainly Microsoft software. Watch this product.

David Jones works for Enex TestLab, one of Australia's most experienced, independent technology test facilities. After more than 16 years with RMIT IT TestLab, Enex's founders acquired the business from RMIT in 2005.