New virtualization benchmark for clouds and datacenters

Everyone uses virtualization, but almost no one talks about benchmarking it. It's easy to understand why. It's hard to do. Fortunately, the Standard Performance Evaluation Corporation (SPEC), recently released a new benchmark for datacenter-level virtualization, SPECvirt_sc2013.

Kunal Singh Sodhi, an IBM content manager, explained that we needed a new virtualization benchmark because its predecessor, SPECvirt_sc2010, which had been SPEC's first virtual server benchmark, had "a good representative workload, but like many benchmarks, its relevancy can erode over time as technology and users' actual workloads change."

So, the new model SPEC virtualization benchmark is designed to improve on the old one. As before, the new  benchmark  is meant to measure the end-to-end performance of all system components including the hardware, virtualization platform, the virtualized guest operating systems, and application software. It's built to support both hardware and operating system virtualization.

SPECvirt_sc2013 builds on SPECvirt_sc2010's benchmark architecture. SPEC explains that, "since SPECvirt_sc2010 was developed, many companies are virtualizing significantly heavier workloads than those in the first generation benchmark. In SPECvirt_sc2013, we are providing heavier and more bursty workloads that require virtual machines with more memory and vCPUs and further stress the system's ability to meet the benchmark's quality of service (QoS) requirements."

According to Sodhi what the changes add up to is that is in sc_2013:

1. Each virtual machine requires more resources. The workloads are used to simulate either more users or users with higher request rates. The resources a VM may need can be nearly six times compared to sc_2010. This was done to match what's happening in many datacenters: many users are moving their more demanding workloads to be virtualized.

2. Many of the VMs in sc_2013 have much higher variability in the resource usage. The workload levels vary quite a bit more than sc_2010. This requires the hypervisor to react quickly and correctly, providing the right hardware resource to the right VMs at the right time.

3. The idle VM is now a "batch" VM. On sc_2010, some of the VMs were simply idle, to represent some portion of VMs which were available, but not in use. Sc_2013 enhances that scenario by having periods for which the VM is idle, but also has periods that it must handle a batch processing job. You can think of these as servers which may have been idle all day, but perhaps processed "end-of-day" jobs which must be completed before the next day.

The first public results from sc_2013 tests are now available from SPEC. While these are valuable, if you really want to know which virtualization stack will work in your datacenter or on your private cloud the best, you'll need to run SPECvirt_sc2013 for yourself. Its predecessor will be supported until February 2014, but I see little reason to continue to use it.

SPECvirt_sc2013 is available for download on the SPEC website for $3,000 for commercial licensees and $1,500 for academic and eligible non-profit organizations.

Related Stories:

• Yes, virtualization is faster (sometimes) than native hardware
• The Linux Foundation releases Xen 4.3 virtualization manager
• IBM to bring Linux KVM virtualization to its Power server line
• Research: Desktop virtualization growing in popularity
• Dell updates virtualization suite, makes VMware support a priority