Virtual unrealities exposed
Over the past 18 months, the buzz around server virtualisation has gradually evolved to a point where customer demand almost matches the hype from vendors. Yet the technology really is nothing new, as anyone familiar with Unix and mainframes will point out — mainframes were running virtual partitions in the 1970s. So, where exactly is all the momentum coming from?
Virtualisation is a broad term, taking in the virtualisation of input/output (I/O), storage and much else, but the recent interest has been focused on virtualisation of servers — essentially the ability to run more than one operating system on a single piece of hardware. While the technology for this is essentially old hat, it is relatively new to the x86 platform, with VMware — currently dominant in x86 virtualisation — getting into the game in 1998.
It's only over the past few months, however, that virtualisation has really taken off on the x86 world, due to a few different factors — server replacement cycles coming around, more mature management tools for virtualised servers, support for a host of new features such as symmetric multiprocessing (SMP), where two or more processors are connected to a single memory, and growing awareness of how well virtualisation works.
One result is that VMware (owned by EMC since the end of 2003) recently broke the symbolic barrier of $100m in revenues per quarter, a figure that has been doubling year-on-year. In February a Forrester survey of 1,221 North-American companies with more than 1,000 employees found that 41 percent were already using virtualisation or were planning pilot tests; overall, 60 percent said they would spend money on virtualisation over the next 12 months. Forty-three percent said they considered VMware most often for x86 server virtualisation, compared with 24 percent for Windows Virtual Server.
Find out more about the main companies providing virtualisation technology
VMware is unlikely to monopolise this cash cow for long, however, with Microsoft and others keen to push their alternatives. Microsoft's Virtual Server is probably the least mature of the lot, but it is improving rapidly, and the software giant is typically giving the technology away. Moreover, virtualisation will be built into Longhorn Server within months of the operating system's release.
The other potentially disruptive development is the emergence of an open source virtualisation project that seems to have nearly everyone rallying behind it. Xen, which uses a different underlying approach from Microsoft and VMware, is built into the latest Linux distributions from Red Hat, Novell and others, and is getting Solaris support within months.
A more flexible approach
Virtualisation is essentially about abstracting resources such as computing power, storage or applications, creating flexibility in the way the resources are used. VMware was the virtualisation pioneer in the x86 realm, and is largely responsible for its current popularity on x86 servers, a fact even competitors readily admit. "VMware discovered the server virtualisation market. They educated that market for everybody," says Simon Crosby, chief technology officer of XenSource, the commercial software company founded by Xen's creators.
In the operating system virtualisation realm there are several different approaches, which can be categorised by the degree of virtualisation they offer. Software such as the PowerPC version of Virtual PC, for example, emulates the hardware platform completely in software, letting you run an operating system designed for…
…entirely different hardware. This entails a big performance hit, however. VMware and Microsoft's Virtual Server, by contrast, only virtualise enough of the hardware to allow operating systems to run in isolation, but the operating system has to be designed for the hardware it's running on. Such an approach still entails a significant processing overhead, however.
Some approaches, such as Solaris Containers, BSD jails and SWsoft's Virtuozzo, virtualise at the operating system level, reducing the load significantly. Only one type of operating system can run on a single physical machine, and all the operating system instances use the same kernel. For putting up with this limitation, the benefits are low overhead, improved performance and massive scalability — potentially hundreds of server instances per machine.
The approach taken by Xen, called paravirtualisation, doesn't simulate hardware at all, instead offering an API that gives each operating system direct hardware access. This means very little overhead and the associated performance improvements, but requires modifications to the operating system. That's not a problem for an open source OS, but poses more of a problem for the likes of Windows.
The current introduction of virtualisation support in hardware — Intel's Virtualization Technology (VT) and AMD's Secure Virtual Machine (SVM) — allows Windows to run under a paravirtualising "hypervisor" such as Xen without modification.
So what's the appeal, exactly? Back in the late 1990s, VMware's customers initially found virtualisation a useful way to build particular virtual environments for testing applications, or for testing software patches before deploying them on production systems.
Around 2001 — amid budget cutbacks, and in the wake of the late-1990s server glut — users began getting seriously interested in consolidation for their production servers. (Sun estimates that most production servers are running at about 15 percent utilisation). The idea was that "server sprawl" could be brought under control, and processing capacity used more efficiently, by loading a number of independent servers onto a single system. This was part of a broader trend towards "utility computing", which encompasses the idea of linking large numbers of heterogeneous servers together into a single pool of resources that virtualisation can divide up at will.
"Fundamentally, we've been approaching a crisis point in the amount of complexity, ever since the birth of client-server," says analyst Gary Barnett of Ovum. "Since the turn of the millennium, we've been reaping the results of our over-exuberance, in buying all these little bits of heterogeneous technology. People have been saying, 'Let's get this under control,' and virtualisation is one of the tools key to doing that, alongside things like clustering."
Advanced uses
According to VMware's view of things, companies have now moved on from simple clustering services to what it calls infrastructure virtualisation. This includes a range of advanced management services, each of which VMware says have seen significant customer takeup.
One is disaster recovery, in essence the capability of automatically shifting a running server from one machine to another, with little or no interruption, in the case of hardware failure. Most server virtualisation offerings…
… now provide management software that can do this automatically, with no break in service. Virtualisation also makes it much easier to back up an entire data centre; VMware says one customer, with 200 virtual machines running in a centre, can have a backup running in 20 minutes. VMware says two-thirds of its customers are using virtualisation for disaster recovery."Previously, if you wanted disaster recovery, the hardware, operating system and applications were all tightly linked to each other, so in your secondary data centre you had to have the same hardware, applications and the rest, and keep them all in sync," says Raghu Raghuram, VMware's vice president for data centre and desktop platform products. "With a virtual machine, you can take that file, transport it by SAN or tape or what have you, to another data centre, and get it up and running immediately".
Shifting servers from one machine to another at will makes it easier to carry out hardware maintenance and load balancing. VMware says more than half of its customers are using a tool called VMotion for such services. Other advanced applications gaining momentum among VMware users include rapid provisioning of applications and virtualisation of desktop operating systems, which some companies prefer for the greater security it provides.
Newcomers: Microsoft and Xen
VMware is keen to point out such trends, as it believes competitors such as Microsoft and Xen are currently nowhere near it when it comes to management tools. Microsoft is just getting started in the field. And while Xen's hypervisor has reached a certain level of solidity — it's now launched version 3.0 after more than three years of development — there aren't yet Xen-based tools comparable with those found in VMware's just-launched Virtual Infrastructure 3. "At the end of the day, Microsoft and Xen are playing catch-up," says Gartner's Phil Dawson.
The shape of the virtualisation market is changing rapidly, however, with an unmistakable trend toward universally available virtualisation built into the operating system. Microsoft's latest offering, Virtual Server 2005 R2, running on Windows Server 2003, is available for free — a price that's somewhat difficult to argue with.Virtual Machine Manager, a management tool based on Virtual Server 2005 R2, is set for availability later this year. Most importantly, Microsoft is planning to build a paravirtualising hypervisor into Longhorn Server. "Virtualisation is now commoditised," says Alfred Biehler, product manager for management and virtualisation at Microsoft UK.
Xen is being built into several Linux distributions. Red Hat Enterprise Linux (RHEL) 5 will ship with Xen integrated by the end of this year. Suse Linux Enterprise 10, planned for July 2006, will also integrate Xen. Sun is planning to support Xen in OpenSolaris this autumn, and its Solaris 10 version of Unix will get Xen support in the first half of next year.
"Virtualisation is going to be in the next release of every OS. It's game over," says XenSource's Crosby. "The question for VMware is, what the heck do they do about this? At the moment, we're no more than a really annoying gnat to them, but Microsoft is a truck coming full bore with its headlights on."
Gartner's Dawson agrees. "As soon as Microsoft launches their hypervisor, they're going to flood the market."
Industry observers agree that built-in hypervisors are likely to be a significant threat to VMware's bread-and butter business. But that doesn't mean companies should start pulling the plug on their VMware installations.
"Operating system vendors want to see virtualisation as a feature, but to the enterprise customers using VMware, that's not always the case," says RedMonk analyst James Governor. "It's a strategic platform to many of them. You can see that if you look at the job ads from many of the financial services companies — they're not asking for Windows skills or Linux skills, they're asking for VMware skills. Organisations are mature now in their use of Windows and Linux, and they're looking for more effective ways to run those on their boxes."
VMware has responded to the emergence of free competitors by dropping charges for some of its more basic software and specifications; VMware Server and Player are both free, for instance…
… and its Virtual Machine Disk Format specification is now available without licence. It has also begun buying companies to bulk up its portfolio of management and administration tools, beginning with Akimbi in June. Akimbi makes tools for virtual lab automation, configuration management and self-service provisioning.Do you need it?
Virtualisation may soon be everywhere, but that doesn't mean it's for everyone. There are inherent limitations to the technology — such as the overhead involved — which make it unsuitable for workloads with heavy, continuous processing or I/O demands.
Microsoft's Biehler points to four key uses for virtualisation: server consolidation in the data centre and branch office; consolidating and re-hosting legacy applications; automating and consolidating software test and development environments; and simplifying disaster recovery planning. "If your requirements are one of those four, it's worth considering virtualisation. If not, it's worth first validating whether it really makes sense to virtualise."
More important, though, is to remember that virtualisation is just one means of achieving infrastructure that's reliable, adaptable, doesn't cost a lot and is easy to manage. Virtualisation may be a part of the way to reach that goal, but many companies may find they really just need to address more basic problems.
"When you move to a virtual environment, that doesn't mean it's more efficient," says Gartner's Dawson. "If you take a pile of rubbish and you refine it, you get refined rubbish. People have to learn that they need to discover their portfolios before they virtualise them. The right approach may be to send it off and not touch it."
Also the level of management discipline found in x86-based data centres is often lacking, compared with that in the mainframe and mid-range worlds, says Ovum's Barnett. "If the environment is subject to little or no management discipline, if you don't have that governance and those processes in place, then actually, virtualisation and clustering are probably not going to help you. You need to sort the mess out," he says.
Just moving to network-based storage may boost reliability so that companies may have little incentive to go further, Barnett says.
That said, the benefits of virtualisation will almost certainly ensure its rapid growth for the near future. "VMware now has references for companies using the technology on a decent scale. They can point to people really doing it and benefiting from it," Barnett says. "That is always going to result in an acceleration of usage and adoption. We expect interest in virtualisation to keep accelerating."...
(See the next page for a overview of the leading players in virtualisation)
Who are the main companies providing virtualisation technology and what are there different strategies?
1. VMware: The old hand
VMware dominates virtualisation for x86 systems, and more or less invented the market others are now trying to cash in on. Three of its core products are VMware Workstation and the freely available VMware Server (formerly GSX Server) and VMware Player. The software runs on Windows or Linux.
Workstation is designed for applications such as software development and testing, while Server is targeted at the data centre, for uses such as testing patches, simplifying provisioning and using virtual appliances. VMware Server was released in July, replacing ESX Server. VMware Player runs virtual machines created by other applications — including Microsoft's VirtualPC — but can't create its own.
Those products need a host operating system to run on, while VMware's flagship ESX Server drastically increases performance by eliminating the need for a host. Instead, ESX uses its own stripped-down kernel, and runs on bare-metal servers. The approach allows support for more guest virtual machines, but the ESX kernel doesn't have as broad support for hardware.
VMware Infrastructure 3, released in June, includes ESX Server, VirtualCenter management tools, Virtual SMP (symmetric multi-processing) up to four-way, VMotion, VMFS distributed file system software, and new tools such as Distributed Resource Scheduler and High Availability and Consolidated Backup. VMotion allows guest virtual machines to be transferred from one physical system to another without interruption, and can be used for things such as hardware maintenance and load balancing.
"Companies are no longer treating servers as a single entity, they're treating them as part of a giant resource pool," says VMware's Raghuram.
Competitors such as Microsoft and Xen may have achieved a significant foothold in the types of services provided by VMware Server and VMware Workstation, but may not be able to offer anything comparable to the mature administration tools of VirtualCenter for some time. "The ambitions of people like Microsoft and Xen go beyond basic virtualisation, but they have a long way to go," says Raghuram.
2. Microsoft: Grabbing a slice
Microsoft offers Virtual PC — acquired along with Connectix — as its desktop virtualisation tool for Macintosh and Windows. Virtual Server, which requires Windows Server 2003, was also developed by Connectix, but hadn't yet been released when the company was acquired.
Virtual Server 2005 R2, the latest version, introduces support for Linux guest operating systems, the ability to make use of (but not virtualise) SMP, support for x86-64 hosts (but not guests), and a redesigned Web administration interface. Virtual Server needs a host OS, unlike VMware's ESX Server or XenSource's Xen Enterprise, meaning significant processing power overhead.
Virtual Server has been demonstrated with support for Intel's VT and AMD's SVM, which will be included in the next release. Microsoft is currently beta-testing Service Pack 1 for Virtual Server 2005 R2, with a Beta 2 scheduled for Q4 2006. Also in development is System Center Virtual Machine Manager, Microsoft's answer to VMware's VirtualCenter, with a release scheduled for this year. The tools will only support Windows guest OSs, leaving Linux support to third parties. In a classic case of Microsoft's thinking on integration, all of Virtual Server's administration tools require technologies like Internet Explorer and Active Directory.
As of April 2006, Virtual Server is available free of charge. Microsoft also made Virtual PC a free download in July, at the same time announcing a that Windows Vista Enterprise's licensing terms will allow customers to install up to four virtual copies of the OS for a single user on a single device. The licence doesn't require use of Microsoft's technology to create the virtual machine.
Most importantly, Longhorn Server — currently scheduled for 2007 — will get a built-in hypervisor (code-named Viridian) within about three months of release. The strategy is clear: virtualisation, in Microsoft's world at least, will become an operating system feature, just as the Web browser did.
Reviewers have not been overly impressed with Virtual Server so far, but Microsoft believes it's good enough for most uses. "Some customers have niche requirements that might require niche products," says Alfred Biehler, Microsoft UK product manager for management and virtualisation. "For the majority of the market, we believe we've got a good solution. And it will just get better." Microsoft also argues there's an advantage in having the entire operating system and virtualisation stack supported by a single vendor.
Microsoft has been paying attention to interoperability issues, say competitors such as Xen. Viridian will use paravirtualisation, and architecturally is very similar to Xen, which should make interoperability easier, according to Simon Crosby, chief technology officer at XenSource.
In June 2005 Microsoft made the specification for the Virtual Hard Disk (VHD) image format (used by Virtual PC 2004 and Virtual Server 2005) available under a royalty-free licence, to help it compete with VMware's proprietary equivalent. "Microsoft has absolutely got the interoperability message," says Crosby.
3. Xen: Old head on new shoulders
On the face of it, Xen may seem like a newcomer, but the software has been evolving for nearly four years, and has impressed reviewers with its solidity. Moreover, its open source approach (it is licensed under the GPL) has helped it toward broad industry support.
The support of Linux distributors has been key; Xen will be built into versions of Red Hat Enterprise Linux and Suse Linux Enterprise Server arriving this year, among other distributions. With this integration comes automatic certification on all the hardware that those distributions run on, according to Simon Crosby, chief technology officer at XenSource, the commercial company founded by Xen's initial developers. Sun will add Xen support to OpenSolaris this year, and it will arrive for Solaris 10 next year.
The same dynamic has meant that Intel and AMD have both rushed to contribute to the project, meaning it already supports both companies' hardware-based virtualisation, well in advance of Microsoft. (VMware also works closely with Intel and AMD and supports Intel's VT and AMD's SVM.) "Intel has been a major contributor with VT, and that means AMD contributes, because they can't afford not to," says Crosby.
Xen already beats VMware on some basic virtualisation features, for instance supporting up to 64-way SMP, to VMware's 4-way limit, Crosby points out.
Such accomplishments explain why people take Xen's ambitions seriously. "We are surrounding ourselves with an ecosystem of players who are economically motivated to work with us," says Crosby. Xen itself only carries out the virtualisation part of the puzzle, but its universal availability creates a huge, wide-open opportunity for partners to step in with virtualisation-aware products to handle more advanced management and administration tasks, Xen developers argue.
XenSource itself will be just one of these, with products such as Xen Enterprise, a "bare-metal" answer to VMware's ESX Server. In April Virtual Iron launched version 3 of its virtualisation platform, scrapping its own virtualisation engine in favour of Xen.
Crosby argues that Xen's open approach is its main advantage over VMware. VMWare has an incentive to own the market for tools based on its platform, and its control of that platform means it can kill off competitors at will, Crosby says. By contrast, XenSource says it isn't aiming to own the Xen ecosystem.
Analysts agree this is a big selling point. "Some of the big players may be beginning to feel they've ceded too much control to VMware now. They're nervous about it as a control point," says RedMonk analyst James Governor. But whatever its momentum, Xen has so far barely scratched the surface of the marketplace — it didn't even register in a survey of large North-American businesses carried out by Forrester earlier this year. "Certainly Xen is nowhere near as mainstream as VMware yet," says Governor.