Wanted: Trustworthy Test Data

Users lack the acumen to conduct a proper test

User experience is a function of the peculiarities of their environment and may not accurately reflect product capabilities

The unchecked proliferation of user-generated stats runs the risk of producing results that make life difficult for salespeople, even an improperly derived performance number sticks in the minds of prospective customers who may have heard it.

The third point is less frequently expressed publicly, but you often hear it when you chat with salespeople. Bad test reports stick in the customer’s mind even if the number of good test reports is overwhelming. It is like the old story about the woman who goes to the dinner party, receives numerous compliments on her dress, as well as one criticism. Yes, it is the criticism that she remembers. Unkind user reports, salespeople complain, carry an inordinate amount of weight that often have less to do with the credibility of the report itself than with a more fundamental mistrust of vendors in this post-Enron era. It makes their lives more difficult.That’s the private-speak. The public speak is that users generally don’t know how to test or lack the resources to test properly, or they are describing product performance within a very narrow range of conditions, namely, the workload and infrastructure and managerial processes that exist in their own environments.

The arguments have some merit, of course. Shoddy testing produces shoddy results. However, in can also be argued that shoddy test results are better than no performance testing data at all, which is what many consumers confront today.

It may be the case, as some "large enterprise-class" product vendors have suggested, that products aimed at the small to medium sized company are more suited to end user testing and reportage than more complex wares. These boxes are delivered as "plug-and-play" solutions. If you were to uncrate an array from vendor A, test it with your application to measure I/O rates and throughput, then repeat the process with products from vendor B, C, etc., you might obtain an apples-to-apples comparison worth consideration. The vendor who suggested this to me went on to note that most operating environments and applications in the SME are the same: Microsoft with maybe some Oracle or Linux. At the enterprise level, he suggested, it is a very different reality.

EMC DMX, HDS TagmaStore, IBM Shark, etc. are much more complex animals. They need to be configured and tuned on install, usually at an additional expense, to realize the performance characteristics previously specified by the consumer. These are not tinker toys that can be readily compared out of the box one to another, so the vendors want to restrict the user from sharing any half-baked comparative data.

Interestingly, in this argument are the seeds of its own undoing. After vendor X has taken money to pre-qualify the network or fabric into which the storage is being deployed, and has taken more money to configure the equipment per the RFP to deliver the performance promised to the consumer at time of purchase, why shouldn’t the consumer be permitted to run tests and to compare results to other products that have been similarly pre-qualified, deployed and tuned? Isn’t that apples-to-apples also?When asked this, vendors will often meander into a complex discussion of the types of measurements being taken, what is being measured, where the data is being collected, etc. This is all interesting, but it is off the point.

In the final analysis, the useful thing about user testing is that it is a measurement not only of the platform's performance, but of the vendor's skills and knowledge in pre-qualifying the install and performing the actual configuration of the gear. At the enterprise level, if the performance of the gear is sub-par, then somebody is selling snake oil and the whole world ought to know about it.

The bottom line is that vendor restrictions on the publication of "unsanctioned" user performance tests are indefensible on any rational basis. Interestingly, such restrictions seem to be imposed only by top-tier name brand vendors on their consumer base: Startups tend to favor a much more open and unfettered exchange of test information. This, in turn, reinforces consumer mistrust of brand name vendors.

Unseen Hand With Unseen Performance Data

There are, of course, a few caveats in the argument above. For one, users have not always done a great job of developing functional specifications and may hold the vendor accountable for poor performance even when it is related to poorly documented requirements. Vendors are rightly dismayed to get a poor report from users who are using tests to express their disappointment about a product that failed to support unstated or poorly defined needs.Furthermore, vendors are correct to be concerned about proxy data: Other vendors masquerading as consumers and flooding information outlets with phony test results. This is done even in the presence of license and warranty restrictions and is usually monitored and reported by the offended party.

However, restricting the flow of user-performed test information is a kind of censorship that should not be tolerated by the consumer. It can be prevented relatively easily, by having the provision stricken from the sales contract at the time of purchase. Most sales droids will delete the provision if it is a potential deal breaker and many vendor accounting offices will turn a blind eye to the contract modification because of the priority placed on quarterly earnings in an unforgiving market.

Truth be told, without consumer reportage on the performance of gear, there is no way to police the storage industry. Vendors should facilitate testing, to make sure that it is done properly; they should not attempt to censor it.

One way or another, Smith’s unseen hand will work its way. Either products will ascend to the top of the market on the basis of their own technical merit, or consumers will move away from vendors who use draconian means to ensure that test data is not published.

The second path is a messy one: Until some sort of objective testing produces measurable differentiators between products (or confirms products to be of equal capability), every time a vendor claims that its product is better than its competitors on a price/performance basis, some troublemaker (like yours truly) will hold their feet to the fire to substantiate the claim. "Never trust nobody" will remain the axiom of the Wild West World of Storage.— Jon William Toigo, Contributing Editor, Byte and Switch

Ever done any of your own internal product testing or benchmarking, and shared it? Write Toigo and let him know the net effect.

Organizations mentioned in this article: