Principled Technologies releases first true cloud performance benchmarks

Benchmarking experts take on the difficult problem of measuring Infrastructure-as-a-Service cloud performance.
Written by Steven Vaughan-Nichols, Senior Contributing Editor

We all use the cloud, but measuring cloud performance is difficult. I should know. I've written benchmarks, and I benchmarked the major public clouds for PC Magazine using Primate Labs' Geekbench 3 and other programs, which, frankly, weren't ideal for testing clouds. But now Principled Technologies and the BenchmarkXPRT Development Community have released the CloudXPRT Preview, a free benchmark, which seeks to accurately measure Infrastructure as a Service (IaaS) cloud performance. 

The beta CloudXPRT program includes web microservices and data analytics workloads. Testers can use metrics from both workloads to compare IaaS stack -- both hardware and software -- performance. Armed with this information, you can decide which cloud provider and which stack is right for you.

In the web microservices workload, a simulated user logs in to a web application, which provides: 

  • A selection of stock options. 
  • Performs Monte-Carlo simulations with those stocks
  • Presents the virtual user with interesting stock options

The workload reports performance in transactions per second, which testers can use to directly compare IaaS stacks and to evaluate whether a particular configuration can meet their needs and service-level agreement (SLA) thresholds.

The data analytics workload calculates XGBoost machine learning (ML) model training time. This is a gradient-boosting framework for ML-based regression and classification problems. The purpose of the workload in CloudXPRT is to evaluate how well an IaaS stack enables XGBoost to speed and optimize model training. The workload reports latency and throughput rates.

CloudXPRT requires you to run your tests with the following configuration: 

  • Test systems must run Ubuntu 18.04.
  • You must use Docker containers managed by Kubernetes.
  • The benchmark requires relatively high-end servers. For functional testing, physical nodes or VMs under test must meet the following minimum specifications: 16 logical or virtual CPUs; 8GB of RAM; 10GB of available disk space (20GB for the Data Analytics workload).

CloudXPRT can be configured to run on your local data center, Amazon Web Services (AWS), Google Cloud Platform, or Microsoft Azure.

It is not easy measuring public cloud performance. So much depends on exactly which cloud configuration you use and your workloads; it's hard to make objective judgments. And almost none of the existing server and system benchmarks are up to the cloud challenge. 

As Bill Catchings, co-founder of Principled Technologies and benchmarking expert, explained, "I don't currently see a direct competitor to CloudXPRT. It seems that folks are mostly using old benchmarks in new settings with varying levels of appropriateness. You certainly can run a bunch of SPEC CPU workloads on a cloud, but…"

Catchings added, "Existing datacenter benchmarks don't make it easy to understand how applications will perform on a given IaaS infrastructure. CloudXPRT uses cloud-native components on a hardware and software stack to provide end-to-end performance metrics that allow users to choose the best configuration for their business."

That said, ThousandEyes, a cloud analysis company, does provide a useful annual performance rating for public cloud providers' global network performance.

CloudXPRT's purpose is to, said Catchings, "help fill that void between performance testing using your actual applications and using some existing benchmarks created for another goal. One of the areas where we are still trying to improve is making things work more easily on the different Cloud Service Providers (CSP)s."

Still, this is a first attempt. But he's hopeful that it will both be helpful in its first public release, and with the help of others, become even more useful soon. As Catching said, "I don't think a benchmark needs to be as large an undertaking as it once was. The ability to use existing, open-source code and higher-level interfaces simplifies things significantly. Our hope is that once CloudXPRT gets out there we will get contributions of either code or suggestions from OEMs and others in the industry."

Users, the tech press, vendors, and other testers are welcome to freely use CloudXPRT and help shape the future of the benchmark with their feedback. Testers may also freely access the CloudXPRT source code.

I have high hopes for CloudXPRT. With companies spending a record $107 billion for cloud computing infrastructure services in 2019, up 37% from the previous year, according to analyst firm Canalys, companies want to know if they're getting the most bang from their buck. Strong, objective benchmarks are clearly needed. CloudXPRT is the best effort I've seen to date to deliver that data to enterprises. 

Related Stories:

Editorial standards