Poker, Cheating & Storage Benchmarks

By Paul Travis, January 16, 2009 2:45 PM

Normally, there isn't a connection between poker, cheating, and storage benchmarks. However, I had to laugh this week when I was reading the message boards on Byte and Switch and saw that one of the posters compared cheating at poker to storage benchmarks to make an interesting point. Somebody identified as "magoo75" was responding in a thread about a press release from Pillar Data Systems Inc. bragging that its Pillar Axiom 600 storage system had scored greats results in terms of performance and price-performance on the Storage Performance Council SPC-1 Benchmark Result.

Pillar said its Axiom 600 system beat out comparable products from EMC Corp. (NYSE: EMC), IBM Corp. (NYSE: IBM), and NetApp Inc. (Nasdaq: NTAP), achieving a SPC-1 performance result of 64,992.77 SPC-1 IOPS with a SPC-1 price-performance result of $8.79/SPC-1 IOPS. Pillar said the numbers represent the most cost-effective SPC-1 result for business-class storage arrays.

We have some cynical IT folks reading and posting messages on Byte and Switch, so it shouldn't come as a surprise that somebody called benchmark numbers a phony representation of real world performance. Magoo75 didn't disagree, but made an interesting point -- and this is where cheating at poker comes in. The post states:

For those of you who don't play cards, there should be only one ace of spades in a deck of cards. So the presence of several in a card game indicates something is seriously wrong and there is cheating taking place. In an old Western movie, it would mean somebody is going to get shot.

But back to storage benchmarks and cheating. Does magoo75 make a valid point? I question his logic. In a real poker game, nobody would win the pot if there were three ace of spades in play. As for benchmarks, we've all heard stories about how this vendor or that vendor tuned its chip or computer or software to show better results on one benchmark test or another. As a result, I don't know of too many people in IT that still pay a lot of attention to benchmark results. To be honest, I don't know enough about SPC benchmarks to know whether they can be gamed or whether many people use them to make purchase decisions.

And I am not criticizing Pillar, which has every right to boast about top results on a third-party benchmark test. I would do the same thing. Nor am I suggesting that there was any sort of cheating going on. But the discussion causes me to wonder how much weight people give benchmark results if so many think of them as a game. So I ask you, dear reader: Do you pay attention to benchmark results? Do you consider them valid? Does everybody cheat? Do you use benchmarks them to make buying decisions? Let me know by clicking "Discuss" below or join the discussion on the message board.