NetApp, EMC in Benchmark Brawl

Folks at Network Appliance Inc. (Nasdaq: NTAP) are accusing EMC Corp. (NYSE: EMC)of irresponsible marketing for its latest price/performance claims about its NAS system (see NAS Gateway Boosts EMC).

The NetAppers took offense at EMCs boast that its NS700 recorded a significant price/performance advantage over the entire line of NetApp NAS filers. EMC based its claims on results it achieved using the System File Server (SFS) benchmark supplied by the Standard Performance Evaluation Corp. (SPEC).

Vendors do their own testing for SPEC, and NetApp questions the way EMC achieved its results. NetApp director of technology Keith Brown accuses EMC of using well-known testing tricks to adjust the performance of its systems, and he maintains the price/performance claims are invalid because SPEC does not verify pricing.

In the release, EMC claimed its new NAS700 system delivers a 42 to 55 percent price/performance advantage over NetApp’s systems.

“They’re embarrassing our industry,” Brown says. “That’s deception and misrepresentation of data, and it’s not good for our industry.” [Ed. note: Isn’t all marketing deception?]Brown says EMC’s tricks included turning off its disk data protection features because it drags performance, short-stroking (artificially limiting the amount of disk data the system reads during tests), and souping up the performance of the NS700 gateway by connecting it to two Clariion storage systems.

“Is Clariion performance that poor that they had to use two of them?” Brown gibes.

Does NetApp have a valid complaint? Probably more so on the price issues than the actual performance numbers. The vendors do their own testing, and then the results go through a committee of competing vendors before they are published. NetApp is a member of the committee and so could have asked for a full disclosure from EMC before the results became official. NetApp reps said they posed questions, but did not seek a full disclosure.

“Games can be played,” says analyst Arun Taneja of The Taneja Group. Taneja says EMC could have done all the things Brown charged, but probably disclosed it all in the documentation. “I’m willing to give EMC the benefit on performance from the SPEC benchmark, but the price/performance claims are up for grabs.”

The SPEC only takes performance into account, not price. SPEC president Kaivalya Dixit says that's because pricing varies from deal to deal. So EMC did its own price/performance calculations, and NetApp’s Brown questions whether the prices EMC used reflect the true configuration of the systems tested. An EMC spokesman says they used the most accurate data available.“When making price/performance comparisons, EMC uses the SPEC SFS numbers from apples-to-apples configurations along with estimated list pricing based on several readily available sources, including NetApp's competitive quotes and others,” EMC spokesman Justin Bartinoski says. “If NetApp wants to share their average street price, we'd be happy to recalculate the price/performance comparisons.”

The battle between EMC and NetApp is heating up now that EMC has reclaimed the NAS market-share lead according to the latest IDC

numbers (see Storage Sales Soar Entering '04). Yet skirmishes over benchmark performances are nothing new between these two. Both have been involved in similar spats:

— Dave Raffo, Senior Editor, Byte and Switch