Wireless Testing: 1,2,3

Unfortunately for the vendors, with so few publications doing testing,it's sometimes impossible to coordinate product release schedules withreview opportunities. When that happens and a vendor is looking tocreate a little buzz, it's not uncommon for them to turn to aprofessional test lab for "independent validation." While there are manysuch labs active today, the Tolly Group is widely regarded as the marketleader.

Vendors pay Tolly to perform tests that they know in advance will castthem in a positive light. Tolly employs experienced testers, and thecompany follows a set of published ethical guidelines. The group openlydiscloses its testing methodology for all to see. That should providereaders of Tolly reports with assurance that its evaluators aren'tcooking the numbers. Experienced IT professionals know what they aregetting with a Tolly report: Accurate information presented in a mannerthat is favorable toward the vendor that paid for the test.

However, while Tolly's validation reports provide a service byempirically validating vendor claims, the company crosses the line whenit engages in competitive testing. That's because competitive testing isall about the test plan itself. If Vendor A pays for the test plan, howdo YOU think the test is going to turn out for Vendor B?

Tolly's most recent effort in this area involved a competitive test ofwireless VoIP security commissioned by Aruba Networks and designed tohighlight its product features against market-leader Airespace. Tollyindependently acquired Airespace equipment and, as part of their "fairtesting" process, advised Airespace that the wireless company couldeither actively support the test or not. Airespace executives took onelook at the test plan, which was obviously designed to highlight Aruba'sintegrated firewall capabilities, and said "No way!" In fact, Airespacepointed to restrictions in its license agreement and threatenedlitigation if Tolly tested without Airespace's approval. Tolly backeddown, but published the report anyway--minus competitivetesting--commenting on Airespace's deficiencies throughout the report.The report itself provided little insight into the fundamental issues ofwhether WLAN switches should include integrated firewall capabilities orwhether encryption should be handled in the AP (access point) or theswitch. It was simply an Aruba promotional white paper, the kind ofthing you might expect from a vendor but not an independent test lab.To its credit, Aruba exhibited some moderation in publicizing the test.Yes, the company posted a link to it on its Web home page. But Arubadidn't issue a press release, as many other companies might have done.Still, Aruba should feel a little embarrassed by the report itself,which is what it is. As for Airespace, while the company was certainlyjustified in crying foul about Tolly's methods, its outrage rang alittle hollow. The company commissioned its own Tolly validation reportlast year as a response to competitive sales pressure from Cisco, whichAirespace felt was misrepresenting its VoIP capabilities. When youenlist Tolly to promote your product, you have to live with thatassociation.

The kicker to this story comes from New York Attorney General ElliotSpitzer, arguably the country's leading governmental consumer advocate.Last year, Spitzer's office won an injunction to prevent NetworkAssociates from selling its products with a "speech restriction" thecompany included in its license agreements. The AG's office contestedthe legality of a clause, which purported to prohibit users frompublishing "product reviews" or "benchmark tests" without NetworkAssociates' permission. While Airespace's contract language was not thisexplicit, it would have been interesting to learn whether it would haveheld up in court if Tolly had challenged it.

Dave Molta is Network Computing's senior technology editor. Write to him at [email protected]