Taking terminal servers to task

But one of the toughest problems in implementing them is performance assessment: How much server power do you need to accommodate your users?

My interest was piqued by a recent announcement from VMWare and Citrix talking about the advantages of using Citrix MetaFrame XP in a VMware environment. The idea behind the initiative is that performance of MetaFrame implementations on high-end, multi-CPU servers is much greater when divided into multiple virtual machines (VMs). And there are no additional licensing costs for Citrix since VMs work off a per-user licensing model.

The announcement is timely in that it fits into the server consolidation craze, although from a different angle. Most consolidation arguments revolve around alleged savings in administration and overhead costs, but this one claims an increase in performance.

At a certain level the idea is counter-intuitive. Citrix Metaframe XP is a third-party, internal virtualization of Windows 2000 Server, but would it be efficient to virtualize the system further by running multiple implementations of it under a hypervisor (the term for systems like VMware which host multiple operating systems on a single computer)?

Well, maybe. There are two arguments for this. First, there's the usual stability argument for hypervisors--problems in one instance of a guest OS won't compromise the others. This is true, but it's not the primary argument the vendors are making here, and nobody's saying there's a general problem with stability of Windows 2000 or MetaFrame XP. But they are claiming--and Citrix agrees--that something about combination of Windows 2000/MetaFrame doesn't scale well beyond two to four processors. Second, they make vague allusions to the fact that a single instance of the Windows OS doesn't scale well. It's a plausible argument, based on general experience and conventional wisdom.

The next step in making an argument like this is obvious: You run performance tests. You set up a series of servers with 2, 4, 8, and 16 CPUs, and determine the number of users at which performance in the system degrades with regular MetaFrame, and also with various numbers of concurrent VMs under VMWare. The vendors haven't done this.

Having designed and run many tests like this--both for publications and under private contract--I can attest to the fact that they are difficult and expensive to run, and perhaps VMWare and Citrix just aren't done with their testing. Which brings me back to the point of this column: It's hard to test these things.

There are a number of potential problems. First, it's important to bear in mind how terminal-based systems like this work. All of the important work is done on the server, so if you want to keep the test automation overhead off the server, you're going to need real client systems to test it.

And since it's real client applications that are running on the server, they won't lend themselves to the sort of test automation that makes testing database or mail servers relatively easy. You'll need real, professional testing tools that let you script and automate client applications--tools like WinRunner from Mercury Interactive. In fact, the latest versions of MetaFrame XP add some low-level features specifically to support these and similar tools.

Until fairly recently, things were a lot tougher for folks looking to perform large-scale tests of MetaFrame- and Terminal Services-based installations. Microsoft has some badly documented tools available in the Windows 2000 Server Resource Kit that it built for a large test demonstration of Terminal Services' scaling ability some years ago, but they work very badly. (In some cases they're actually hard-coded to specific server names Microsoft used in the scaling test.)

Most importantly, you have to be careful to understand what you are testing, and the Microsoft scalability test paper does a decent job of helping with this. Different tests and configurations can show the effects of varying CPU, memory, and network capacity, and other variables. It's memory capacity that's usually the most important variable in calculating how many people you can fit on a given server, but the LAN and WAN interfaces can be crucial as well--and the tests that you'd design to measure each separate element (CPU, memory, LAN/WAN) won't necessarily be the same.

The bottom line is that performance claims without performance tests to back them up are incomplete at best, and empty at worst. But good performance testing is difficult, and throwing things like virtual machines into the mix just makes things more difficult. Great. Just what we needed.

Would you consider using MetaFrame in a VMware environment? TalkBack or send e-mail to us.