Hosted virtualisation suites: a group test

A lot of the activity in virtualisation is focused around the datacentre. That's all well and good, but there's a whole world of virtualisation for workstations, where competition for the best suite is red-hot and constantly improving.

Previously we examined the bare-metal hypervisors that are favoured in datacentres. Here we look at Type 2 virtualisation — where one operating system needs to be running to host another operating system.

The line-up for this round-up is: VMware Workstation, Virtualbox, Parallels Workstation and KVM. Wine is also tested to provide some control to the test and because it can achieve some of the benefits of a hypervisor for running Windows applications.

Hardware
The workstation we used for testing was an HP Z800 workstation; this system features dual 3.2GHz Xeon W5580 processors, 2GB of RAM and a 1TB hard drive. A monstrous Nvidia Quadro FX 5800 graphics adapter with 4GB RAM is also part of the mix.

Ubuntu 9.04 was chosen as the base operating system as it provided the widest range of hypervisors to examine and allowed for testing of KVM and Wine, which are not available on any Windows operating systems.

Windows XP SP2 was installed under each hypervisor, and the following tests performed:

• Cinebench single CPU
• Cinebench multiple CPU
• Cinebench openGL
• Start-up time

Windows XP was chosen over Windows Vista or Windows 7 as it is stable, lighter and a well-known quantity. We felt that this would allow each of the hypervisors to show off their potential with the additional extra packages and tools included for XP. KVM was the only hypervisor to not provide an additional package to install to accelerate Windows XP's performance.

Installation
VMware Workstation
Installation for VMware Workstation is straightforward. After downloading the product from its product page, it was a simple case of installing the packages needed in Ubuntu, executing the downloaded software bundle and following the prompts. The bundle is an easier alternative to converting the rpm file for non-rpm-based distributions such as Ubuntu.

The hardest part of this install was having to sign up to the VMware site and receive a trial key, which is a sign of the ease of installation for this mature product.

It's simple to get Windows XP running under VMware Workstation with the unattended install process.

VirtualBox
Two options exist to get VirtualBox onto a Linux system: via the distribution's package manager or via the VirtualBox site itself.

Based on previous experience with a Gentoo box, the VirtualBox OSE package can be more hassle than it's worth. The pre-packaged version available from the VirtualBox site worked liked a charm. Simply download the appropriate file, click to install it and you will have VirtualBox up and running in no time.

The speed of the pre-packed VirtualBox installation is hard to beat.

Parallels Workstation
Parallels' traditional strength has been on the OS X operating system, and one would truly hope so after the drama needed to get Parallels working properly on Ubuntu. It's easy enough to acquire a copy of Parallels and the necessary trial key from its product page, but careful reading of said page should flash some warning lights.

For instance, the site says 'New! Parallels Workstation now installs on new Linux kernel (2.6.23)' — a kernel that has its origins in 2007 and has since been superseded by six stable releases.

Also 'new' to Parallels Workstation, according to its product page, is support for Ubuntu 7.10 and Fedora 8 — yet again, the state-of-the-art Linux circa late 2007.

The real pain with installing Parallels came after installing the deb archive off the site and finding it impossible to complete the 'parallels-config' command. This was because we used the latest Ubuntu distribution. Thanks to the instructions available here we were able to complete the installation, but this process easily made Parallels Workstation the hardest product to get up and running.

Parallels proved the hardest hypervisor to get up and running under Ubuntu.

KVM
KVM's installation is handled by Ubuntu's apt package management system, however unlike Wine (below), there are a number of other steps needed to complete the installation. Some of the steps require editing of XML files, but the instructions are straightforward and should not create any difficulty.

It's also recommended that you install a graphical tool for KVM, to make virtual machines much easier to manage.

In typical *nix fashion, the Virtual Machine Manager shown here does not come bundled with KVM.

Wine
Wine installation is also handled by Ubuntu's apt package management system. An apt-get install wine is all that is needed to install it, followed by running the winecfg command. Once these two commands are completed, you're ready to roll.

Wine integrates fully with the Ubuntu environment.

Manageability & flexibility
VMware Workstation
Upon creating a new virtual machine for our Windows XP test, we were greeted by a unique VMware feature: an unattended install. This is a great time-saver, as XP has a number of prompts that require input throughout the install. Here, you just enter your Windows product key, a username and a password, and VMWare takes care of the rest.

The control panel for virtual machines is well laid-out and easy to use — once again the maturity of this product shines through. It's also the only hypervisor, besides KVM, to explicitly allow editing of the number of cores that a virtual machine can use.

VMware Workstation has a record/playback feature that allows the reproduction of the machine's state, although this did involve dropping the machine back to executing on one core only. It could be useful for debuggers and testers. It also has the ability to run virtual machines' execution in the background and drop the viewing window.

VMware Workstation's management window.

VirtualBox
VirtualBox focuses on one thing — running desktop-focused virtual machines — and does it very well. It's a no-frills approach that keeps VM management dead simple, but removes the flexibility present in other systems.

One problem we encountered was VirtualBox's inability to boot from a NetBSD 5.0 CD. This was the only hypervisor to fail, and was the only outstanding blemish on VirtualBox's record.

All the options for each VM are kept in a separate window, which keeps the VirtualBox interface clean.

Parallels Workstation
While the other hypervisors tend to display a list of virtual machines created within them, Parallels works with its VMs via files. This means that you have to navigate the filesystem and open the file of the VM you wish to use in the same manner as a word processor application, for example. This makes the interface clunky and feel dated in comparison.

Another problem we had with Parallels is that our Windows XP VM required resetting after powering on to boot properly. It's a minor annoyance, but one that can be quite aggravating over time.

The information on each VM is well presented, but it lacks the flexibility of the VMware Workstation or KVM.

Parallels Workstation does a nice job of showing the important configuration settings.

KVM
KVM takes a traditional Unix approach and separates the execution of the virtual machines from the viewer completely. For example, you can have Windows XP running without a window open showing the desktop; it will just run quietly in the background.

Clearly KVM is aimed at a higher level of use than just workstations, and the design reflects this. Although it's recommended you use the available graphical tools, you can revert to XML or a console interface should you wish.

As the KVM viewers are separate from the server, they can connect to KVM hosts that may be running on other machines.

Virtual machines are easy to create and manage easy with the graphical tools, and the ability to drop back into text files gives KVM lots of flexibility and under-the-hood power. With our setup, we could easily create and deploy many Ubuntu virtual machines, an ability which is more at home in the server room.

Along with VMware Workstation, KVM is the only other product that allows you to specify the number of cores available to the virtual machine, showing its 'beyond-workstation' pedigree.

KVM has a tabbed interface, where the VM can be viewed in the 'Console' tab.

Wine
As a program that simply executes Windows programs, the management is left to the user — the applications can live wherever, and Wine will happily execute them.

The flexibility of the system depends on the application, and its status in the Wine Application Database. Unless the program has Platinum status, expect to have problems and minor annoyances as a bare minimum.

There's no question that Wine is meant to imitate Windows.

Performance
Only VMware Workstation was able to run every one of our hypervisor benchmarks. This speaks volumes for the stability and flexibility of the product.

The results of the tests we performed were:

The key rows are the start-up time and OpenGL score in Cinebench.

VirtualBox and Parallels appear to blitz the competition in the single-CPU test, but all is not as it seems. These two products cannot be configured to restrict the number of cores used, so we suspect that they are both using as many cores as possible to boost performance. This is backed up by the fact that Parallels and VirtualBox appear faster than Wine on a single CPU, which lacks the additional hypervisor and OS overhead, yet Wine smashes their scores on the multiple CPU score: what appears as one CPU to the guest OS may in fact be many processes and/or threads on the host OS. We therefore put more credence into the start-up time as a better reflection of performance.

In general usage, each of these hypervisors feel quick within Windows XP. VirtualBox does stand above the competition though, and is blisteringly quick with OpenGL compared to the other hypervisors.

Cost
The following table summarises the cost of each product (excluding support):

Verdict
Based on our tests and interaction with the product, the only option we would rule out is Parallels Workstation. Even though it's the cheapest of the non-free options, the trouble that it gave and the continual need to reset the VM immediately after powering it up struck it off the list. Some of these issues would immediately be ruled out if you're using Parallels on the Mac OS X platform, where it's much more mature.

If you user a Linux workstation and need to run one or two Windows applications, and you don't need a full-blown virtualisation solution, then Wine is the best way to go. The applications are much quicker than they would be in a hypervisor, but do have the caveat that they may not work exactly as you expect in Windows.

KVM is here as a hosted virtualisation suite, but is generally up against workstation-focused software rather than server software, which is its natural terrain. Due to its lack of performance and problems with graphics rendering, KVM is not recommended for pure workstation purposes. However, if you don't mind tinkering with Linux and need to deploy multiple copies of the host operating system — for example in multiple Ubuntu instances — then KVM could work for you.

In the server room, KVM does find itself up against products more in line with what we covered previously. Even so, considering that it's free, easily available on Linux and can replicate its host OS easily, it definitely has a place at the workstation table. However, it's not the winner.

That leaves two products in line for the title: VMware Workstation and VirtualBox.

The clear tear-away from the pack in terms of performance was VirtualBox. When VirtualBox works it's very good and only has issues on the rarest of occasions. The new 3D acceleration built into the product gives it a leg-up on the competition and the fact that it's free only sweetens the deal. A minor grey cloud on VirtualBox's horizon is the recent purchase of Sun Microsystems, the current developer of VirtualBox, by Oracle. It's unlikely that Oracle will kill the product, but it could be tempted to merge it with its own virtualisation solutions or make it more enterprise-focused.

VMware Workstation is highly configurable and built by a company that knows virtualisation. During testing, it was faultless and the management systems were top notch. Although its performance stats were not the best, the feel of the system is not slow. The ability to record a VM session will definitely find a niche.

What is our final recommendation?

If you must have 100 per cent compatibility, certainty and would like support, then VMware should be your choice.

But provided you're not trying to boot NetBSD 5.0, we would recommend VirtualBox. It's fast, easy to use and has that sweet price point — free.