Which cloud will give you the biggest bang for the buck?
After Amazon Web Services' (AWS) Northern Virginia datacenter went casters up, you might think about switching your Infrastructure-as-a-Service (IaaS) cloud provider. If that's the case, Cloud Spectator has benchmarked 10 of the biggest public cloud Infrastructure-as-a-Service (IaaS) providers based on price and value.
Cloud Spectator benchmarked the 10 biggest IaaS cloud providers in North America for both performance and value.
Cloud Spectator tested 10 of the largest, most well-known public cloud providers with North American datacenters. These were: 1&1, AWS, Microsoft Azure, CenturyLink, DigitalOcean, Dimension Data, Google Cloud Platform, OVH, Rackspace, and IBM/Softlayer, aka Bluemix. They examined the performance of vCPU, memory, and block storage along with cost. This gives users the clouds' CloudSpecs Score.
Cloud
You could try to do it yourself, but as Cloud Spectator remarked in its report, "A lack of transparency in the public cloud IaaS marketplace for performance often leads to misinformation or false assumptions. Users and potential users may be led to view cloud computing as a commodity, differentiated mostly by services. The reality of performance in cloud computing impacts the user differently from Cloud Service Provider (CSP) to CSP, involving everything from the physical hardware (eg, Intel or AMD, SSD or spinning disk) to the cost of the virtualized resources."
The company presents its performance results in two categories: Virtual Machine (VM) Performance and Block Storage Performance. VM Performance tests the VM CPU and memory. This performance data was aggregated into one score including both CPU and memory. Block storage was tested using two different tests.
Value was defined as the ratio of price and performance. You might think all cloud providers are the same but the results vary by more than seven times across the compared IaaS providers.
The relatively obscure 1&1 achieved the highest CloudSpecs Score. This was due to strong VM performance and the most inexpensive packaged pricing. Other certain providers, such as Rackspace, may have achieved above-average performance for VM environments, but their price-performance value came in lower because of higher costs. That said, this study did not account for additional features such as managed services, which are included in Rackspace's pricing.
The benchmarking also showed performance differences, which illustrate a lack of standardization in public cloud IaaS architectures. For example, one service's high-speed vCPU is another's standard vCPU.
In overall VM performance, Microsoft Azure's VMs -- Performance Index score of 92 with 2 percent variability -- showed the highest median performance and lowest performance variability. AWS and Google Compute Engine also showed relatively little performance variability. On the other hand, DigitalOcean's VMs showed the lowest median performance and highest performance variability -- Performance Index score of 60 with 16 percent variability -- in the study.
When it came to block storage performance, Rackspace, powered by OpenStack, turned in the fastest input/output operations per second (IOPS) numbers. At the same time, Azure showed the best performance variability over the testing period.
Dimension Data had both the poorest disk performance and the greatest variability. AWS didn't do much better. This appears to be because AWS throttles storage IOPS. The level of AWS Elastic Block Store (EBS) throttling depends on the size of the disk. That said, AWS' high level of variability is due to a built-in EBS burst function. This is not because it's unstable.
Before deciding which service you'll pin the hopes of your company on, you should know something about Cloud Spectator's methodology. It was designed to capture performance statistics based on synthetic cloud infrastructure performance, which requires steps including extended testing over 24 hours and running multiple VMs in parallel.
Notice the word "synthetic." As Cloud Spectator points out, its results are "not representative of any specific workload. Therefore, the results are used for comparison purposes only, and cannot be applied to predict application performance. For example, on AWS, the gp2 [General Purpose SSD] block volumes demonstrated periods of burst that were limited due to the continuous bursting over the 24-hour period, but the gp2 would not be a recommended option for applications that demand continuously high IOPS performance."
You should also keep in mind that there are no standard industry cloud benchmarks, not even just for IaaS clouds. Cloud Spectator does a good job -- I know, I've benchmarked clouds myself -- but it's not easy and there are many variables to consider.
In other words, this benchmark is a good guideline. But, before deciding on a cloud, you're still going to need to run test projects with your software to find out which one will really fit your needs the best.
Related Stories: