IaaS Performance Benchmarks Part 3: AWS C3

Editor's note: This post was updated Monday with information from Amazon after initial publication.

This is the third part in a series of articles about creating my own IaaS performance benchmarking project. In the first part, I explained the methodology I'm using to test instance types across major IaaS providers. In the second part, I ran benchmarks across every instance type in every U.S. region run by Amazon Web Services (AWS), the most widely used IaaS provider. In this part, I look specifically at the new C3 family of instances that AWS launched at its re:Invent conference, and update my AWS charts accordingly.

Amazon Web Services has been around long enough to want to deprecate/retire some of its instance types, as seen both by the excellent M3 family (replacing the M1 instance types) and now the new C3 family (replacing the C1 instances).

While it appeared to me initially that Elastic Block Storage (EBS) backing was required for the new C3 instances, Amazon informed me that's not the case. In my testing, I used EBS-backed C3s.

Two important notes about these updated benchmarks. First, in order to get these results, you must run C3 instances on Hardware Virtual Machine (HVM) EBS volumes, not ordinary EBS volumes. AWS will let you run on regular EBS volumes, but there is absolutely no reason to do this, because it costs the same to run under HVM EBS volumes, and the performance is significantly better. (For more on this, take a look at my first set of AWS results and my comparison of EBS-backed M3 instances versus HVM EBS-backed M3 instances).

Second, I'm dropping the SysBench results from these and further benchmarks, because, after hearing significant feedback from benchmarking experts, I don’t think they’re adding anything more to these foot-deep, miles-wide analyses across IaaS vendors. Now onto the results.

[Read about a new initiative to develop a framework that provides secure connectivity from any device to a cloud application in "Cloud Security Alliance Launches Secure Network Effort."]

To test the C3 family, I launched 30 AWS VMs -- five new C3 instance types each in six different availability zones (AZs) in US-East and US-Oregon (you can't launch HVM EBS-backed instances in US-West). As I found with the other AWS instance types, the C3 instance types had consistent performance for each instance type.

My main takeaway: the C3 family is pretty fantastic, and really sets a new bar for affordable performance. Many people use the c1.medium and c1.xlarge instances for Web servers and databases, and if you do, switch to the C3 posthaste, as you get significantly better performance for essentially the same price.

View Larger

If you’re running single-threaded, general-purpose applications, you’re crazy if you’re not using the C3 or M3 families. The c3.large, in particular, is really fantastic, at 15 cents per hour on demand. And, in general, unless you can’t afford 15 cents/hour, it’s hard to make a case that you should be running any of the M1 or C1 instances any more.

View Larger

The one-thread-per-core UnixBench results encompass such a large range on AWS that it’s necessary to break them into two charts in order to understand the range across instance types. I have chosen to do this on price, as opposed to performance, but please note that there are instances on the cheaper chart that handily beat instances on the more expensive chart in these UnixBench scores.

View Larger

One interesting note is that the C3 series doesn’t quite scale linearly with respect to these UnixBench scores: The average score jumps from 1867 to 2966 to 4876 to 6924 to 8472, but the price doubles each time. These benchmarks are running on CentOS (we saw previously in this series that running on Ubuntu improved the scores significantly), and I’m using RightScale’s CentOS images to get parity across IaaS vendors, so take the specific scaling here with a grain of salt. However, I would strongly suggest that you test how well they scale for your application, because it may make sense to stick with the c3.large and scale horizontally.

All in all, the C3 family looks excellent in these benchmarks, and if you’re talking to other IaaS vendors that are showing you comparison benchmarks, you need to make sure that they’re doing a comparison against the C3s. Note, though, they almost certainly aren’t -- I haven’t seen a set of benchmarks touted by another IaaS vendor that ventured outside of the deprecated M1 family.

In the next few weeks, I’ll post my results from Google Compute Engine and Windows Azure, and comparing them to the best-performing AWS instance types. And again, please remember the aim of this benchmarking project: I’m trying to give all of us a high-level view on IaaS offerings. I would advise you to do further testing, based upon what you can afford.