IaaS Performance Benchmarks Part 2: AWS

This is the second part in a series of articles about creating my own IaaS performance benchmarking project. In the first part, I explained the methodology I'm using to I'm test instance types across major IaaS providers. In this part, I post the results from my tests of Amazon Web Services instance types.

I launched more than 175 AWS VMs across current instance types, in all three U.S. regions (East, West and Oregon) and in three availability zones (AZs) per region. (The West and Oregon regions have only three AZs; East--for me--has only three AZs that allowed me to launch all of the instance types.) I ran instance-store-backed instances for every instance type I could; I ran EBS-backed for the t1.micro, and HVM-EBS-backed for the m3s and high-computing instance types.

In addition, I ran EBS-backed for a few of the instances that you can run either instance-store-backed or EBS-backed to see if there was a performance difference. The high-computing types aren’t allowed in US-West, and the cg type is only available in US-East. I was blocked from running a few instances in one of the US-West AZs. I ran a mix of spot and on-demand, and couldn’t find any performance difference between them.

I had to patch UnixBench to get it to run properly on the cg1 and cr1 instance types, as both have more than 16 cores. In my charts, I used on-demand pricing to compare relative differences in pricing. When I compare AWS to other providers in later parts of this series, I will discuss both reserved and on-demand pricing for AWS.

I also found that, in at least some circumstances, Ubuntu outperformed CentOS in benchmarks. However, I am sticking with CentOS for all benchmarks because there is more support for it across IaaS providers. (Specific circumstance: m3.2xlarge with HVM EBS AMIs had a single-core UnixBench score that was more than 20% higher on Ubuntu 12.04 than on CentOS 6.4).

Instance-Type Bake Off

Here are the main takeaways that I have from comparing instance types across the benchmarks:

•m1.small and t1.micro are both probably the best instance types as far as price/performance (at least if UnixBench is a reasonable measure of performance for you), but they are poor as far as performance-per-machine goes as compared with other options. If you’re using either, just know that if you pay more, you’ll get much better performance.

•The m3 family did really well in these benchmarks, but be warned that this is performance on HVM EBS AMIs. If you run m3s on regular EBS AMIs (which AWS will let you do, perhaps because HVM AMIs don’t appear to be available in US-West), the benchmark scores are significantly lower (well less than half in my benchmarks). Also, I would recommend testing Ubuntu (and perhaps other flavors of Linux) over CentOS for these, as I saw significantly better scores under Ubuntu with m3 instances in particular.

•Looking at the $1/hour or less (on-demand pricing) options: The m3.2xlarge ($1/hour), c1.xlarge ($0.68/hour) and m3.xlarge ($0.50/hour) stand out with significantly higher scores than the rest. One reason for favoring the c1.xlarge: You can run it backed with Amazon’s instance-store, instead of EBS, which means that you will not have the two single points of failure that exist with EBS-backed instances (the physical machine that the instance is on and the EBS volume).

•The cheapest instance that does fairly well in these benchmarks is the c1.medium ($0.145/hour). You have to pay more than five times as much to get double the performance in many of the benchmarks I ran, and the performance is close to much more expensive instance types for many uses.

•I think it would be hard to justify using the m1.xlarge in almost any circumstance. The m3.xlarge is only 2 cents/hour more than the m1.xlarge (on demand), has the same or better specifications, and gets significantly higher scores in these benchmarks.

•As you spend more money on AWS instances, you get generally better performance in these benchmarks. For example, the cg1.4xlarge costs about 3.5 times more than the c1.xlarge, and produced a multicore UnixBench score of roughly 3.5 times as much as the c1.xlarge.

•Amazon provides a good descriptive guide to picking instance types for particular uses, and these results should not be considered in any way a better general guide to selecting an appropriate instance-type family.

That said, I have met many AWS users who seem to select instance types based on either a “Hey, this one looks good and cheap” or a “Let’s spend as much money as possible so it goes fast” basis. If you know (or suspect) that your AWS deployment choices are based on this kind of criteria, these benchmarks should push you toward re-evaluating your selections.

UnixBench Single thread

UnixBench one thread per core

Note that in both SysBench charts above, I cut off the top for both the m1.small and t1.micro; their results were literally "off the chart."

Next Page: AWS Region Bake Off

In order to compare all three regions, I had to ignore the high-performance computing instance types (which aren’t available in US-West). I also did a straight comparison of US-East to US-Oregon, just of the high-performance instance types that are in both. Finally, I ignored a small number of my 175 runs in order to make sure I had the same number of the same instance types in each region, and kept diversity across AZs. This means that the three-region comparison was over 38 VM runs each, and the two-region, high-performance comparison was over 12 VM runs each.

My takeaway: It’s a wash. It's hard to argue that any region is substantially better than the rest, although if you had to pick one, US-East does better here than the rest.

Now, comparing US-East and US-Oregon (only the high-performance computing instances):

Here, US-East beats US-Oregon fairly convincingly for high-performance types, and US-East also is the exclusive region with the very good cg1.4xlarge instance type. So, I see US-East as the obvious choice for running high-performance instance types.

These cross-region analyses make a decent data-based rebuttal to recent complaints about Amazon having a “legacy” cloud in US-East.

[Find out what security issues to consider when signing a contract with an IaaS provider in "Top IaaS Security Requirements."]

I also looked at cross-AZ performance, but because everyone can have different availability zone assignments (for example, my US-East-1a could potentially be using data centers that are split across your US-East-1b and US-East-1e), charts aren’t particularly useful here. Suffice to say that I did see performance differences across AZs in all regions, so that you’re going to get the luck of the draw, or you’ll have to performance-test everything you launch to get the best performance. You can’t just pick a particular region or availability zone and expect it to be the best.

That said, the performance differences aren’t that huge, so luck-of-the-draw is probably fine for most use cases. It's probably cheaper to pay for a beefier instance than to performance-test everything on load.

Other Observations

I've always wondered whether there was a significant difference between running EBS-backed instances or instance-store-backed instances, especially with respect to I/O; I can imagine theoretical arguments going either way. The short answer is that there isn’t a big difference (at least as far as the benchmarks I ran). I launched 8 c1.xlarges, 8 m1.mediums and 8 m2.xlarges as both EBS-backed and instance-store-backed across all three U.S. regions, and did not find significant differences:

Finally, I think it’s worth acknowledging how good the AWS service is: Can you imagine telling someone 10 years ago that it would be possible to test the performance on more than 175 different VMs in more than nine different data centers across the United States with a few hours of programming, a few hours of actual machine runtime and about $150 in total costs?

I ran into zero problems with AWS. My only minor complaint is that some instance types weren’t runnable in some availability zones, but I was able to run every instance type in each region in which Amazon said it would work, on-demand, with launch times of a few minutes. And a tip of my hat to RightScale as well; its RightImages, ServerTemplates, RightScripts and API made this overall project possible.

During the next few weeks I will post results from other IaaS providers and compare them to AWS and each other. I will likely focus on just a handful of AWS instance types; I just don’t see the point of comparing any of the instance types in the m1 or the more specialized families that don’t show well in generalized benchmarks--like the m2, hi or hs--to other providers, considering that they don’t do well in these benchmarks and in my personal experience. By the end of this project, we’ll all be a bit more informed about the options across IaaS providers.