Why Storage Performance Metrics Mean Little

Every week or so another vendor claims a new performance record or achieves a new level of I/O operations per second (IOPS)--and it is starting to drive me a little crazy. Most of the performance statistics now at least include the configuration of the system that was used for testing. But those test configurations are so far away from what the real data center can afford, or needs, that the performance test results are almost useless.

Most data centers don't have an application that needs 2 million IOPS or have the network infrastructure to back up 10-TBs per hour. Instead, most of the time when I speak with a CIO or IT manager, their goal is to achieve as much performance as they can within a given dollar figure. The budget drives performance level, not the storage system. The major problem with these benchmarks is they show you what the systems can do if the organization has an unlimited IT budget.

As a result, these performance tests that have no regard to cost are immediately discounted by most IT professionals. Their only real value is being able to understand what the system can do at its maximum--and, of course, for vendors to beat their chests. IT professionals expect to run their own performance tests and don't count on the vendor test. The problem with this reality is that it is a waste of everyone's time. The vendors are wasting time bragging about numbers that users don't expect to achieve, and users are wasting time coming up with their own numbers when the vendors are fully capable of providing them.

A performance benchmark of how many IOPS the storage device generates per dollar is probably of more value than how many total IOPS a storage system can produce. The better alternative may be to provide users with performance price bands. For a $50,000 investment you are going to get this, for a $200,000 investment you are going to get this, and so on. They should also clearly state what would need to change in the environment to get those levels of performance. For example if you have a 10-GbE iSCSI array, you can't assume that 10 GbE will be in place; you have to report that you are expecting it.

With this information the IT manager could look at the performance numbers available for their budget band and decide if they need to ask for more budget or just begin final vendor selection. Yes, you should still test the potential solution yourself, but maybe you won't have to test as long or as many products to get to your answer.

Storage Switzerland's commitment is that when we talk to vendors that boast a performance capability, we will ask them to give us the performance number within certain price bands.

Follow Storage Switzerland on Twitter

George Crump is lead analyst of Storage Switzerland, an IT analyst firm focused on the storage and virtualization segments. Storage Switzerland's disclosure statement.