Radio-Tested WLAN Clients

You've just rolled out a minor upgrade of your Wi-Fi access points, and suddenly your notebook users begin calling the helpdesk with "blue screen of death" reports as soon as they power up their systems. The helpdesk managers haven't yet made the connection between the notebook problems and the AP upgrade, but you have. You just can't quite figure out how it could happen.

Jonn Martell and his colleagues at the University of British Columbia (UBC) survived this scenario in December 2005, after the team made some relatively small changes to its Cisco WLAN infrastructure. The problems, which occurred in notebooks with Intel Centrino chipsets, were traced to a Centrino driver problem. Intel resolved the issue, but as Martell, who at the time was UBC's wireless services manager, puts it, "We'd never seen a wired network card BSOD a computer based on the firmware installed on a switch!"

We're not suggesting that problems of this magnitude are an everyday occurrence, but Martell's experience points up the need to consider the balance between Wi-Fi client devices and Wi-Fi infrastructure. Although many technology analysts focus on differences in Wi-Fi infrastructure offerings, few look at the client side of the wireless equation and how the interaction between different clients and APs affects system performance.Our goal in this evaluation was to perform a systematic analysis of the differences between Wi-Fi implementations on leading enterprise-class notebook computer systems.

Products from Dell, Hewlett-Packard, Lenovo (IBM) and Toshiba make up more than 75 percent of the enterprise notebook computer market. Each company sent us one or more systems. We let each vendor choose a model it felt best met the needs of mobile enterprise professionals, but we also made some specific requests. First, we wanted systems equipped with dual-band 802.11 a/b/g radio modules, which complement the trend toward dual-band (2.4 GHz and 5 GHz) enterprise WLAN infrastructure. Second, because we were interested in evaluating differences not only in notebook systems but also in embedded radio modules, we asked each vendor to supply both a Centrino and non-Centrino wireless offering.

Dell sent us two Latitude 410 units, ultralight notebooks that weigh about four pounds each. One system came with Intel's Pro/Wirelesss 2915 mini-PCI radio module, while the other had a Broadcom 4318 radio.

Lenovo sent us two IBM ThinkPad T43 5.5-pound notebooks--the company's most popular enterprise-class systems. One of the T43s came with an Intel 2915 radio, while the other came with an Atheros AR5211 radio.

HP sent us a Compaq NC 4200, a 4-pound notebook comparable in size to the Dell offering. HP shipped us the system with an Intel ProWirelesss 2915 radio installed but also sent us a Broadcom 4318 radio module, which we installed as a replacement for the Intel module when conducting our Broadcom testing.At slightly more than 6 pounds, Toshiba's Tecra S3-S411 TD was the largest notebook we tested. Toshiba supplied a system equipped with an Intel Pro/Wirelesss 2915 radio, but did not provide an alternative radio.

We tested the systems in our Syracuse University Real-World Labs®, running them through a range of bench tests using RF test equipment from Azimuth systems, as well as field testing them in a newly renovated building to evaluate performance, transmission range, roaming and battery life. We noted significant, though not dramatic, differences among the notebook offerings and among the radio modules.

Although the differences were not earth-shattering, the ThinkPad T43 with the Atheros radio module offered the best wireless experience, both for 11g and 11a. Lenovo's client utilities were the best of those we evaluated, offering a wide range of features and consistency across radio platforms. Systems with Intel's Pro/Wireless adapters also fared well, but trailed the ThinkPad-Atheros combination by a modest amount in every measure. Although the Broadcom-based systems performed acceptably, they did not measure up to Atheros or Intel in performance and range when using 802.11a. However, when combined with the antenna subsystem on the Dell notebook, the Broadcom-based system turned in the best overall performance for 802.11g in our field testing. Finally, while we give Toshiba some credit for being the only vendor to provide a physical on-off switch for its Wi-Fi adapter, the Tecra lagged others by a significant margin in both performance and range.

Aren't They All the Same?

All the Wi-Fi silicon vendors represented in this evaluation--Atheros, Broadcom and Intel--have gone through multiple generations of chip development, and all provide credible product offerings. Still, there are important differences among them. Atheros' superior rate-versus-range performance when operating in 802.11a mode, for example, might be a factor for enterprises committed to scalable dual-band Wi-Fi deployments.

In the same way that the chip vendor offerings have matured, so have the notebook systems. Wi-Fi is no longer a bolt-on afterthought. Rather, vendors have improved system designs with better antenna technology and enhanced the isolation of radio modules from internal system interference.

The quality of a product's Wi-Fi subsystem--radio module, antenna and software--is increasingly viewed as a potential differentiator. Our experience suggests that all the vendors have made great strides in recent years, providing improved performance and more flexibility. All the systems included in this evaluation came to us with support for a broad range of 802.1X EAP (Extensible Authentication Protocol) types, for example, which is important for enterprise deployments that link encryption keys to RADIUS-based authentication.

Test Results

We evaluated rate-versus-range performance of all the notebooks and their different radio modules in two test environments (see "How We Tested Wi-Fi Clients"). The first was a controlled RF environment that isolated each system and its radio module from the antenna. The second was a real-world office environment in which we ran performance tests in five locations.

"Azimuth Rate Versus Range 802.11g," right, shows the results of a rate-versus-range test performed on the Azimuth Systems W-Series WLAN Test Platform, with throughput measurements shown in 1-dB attenuation increments. The use of decibels as a measure of signal level is common in the industry, and the challenge of radio designers is to develop systems that can maintain a good link even when confronted with very faint radio signals. A decrease of 3 dB equates to a 50 percent reduction in signal level. Negative dB values represent progressively smaller fractions of one milliwatt, which is the industry-standard reference level for these devices.

Although there are some differences among 802.11g systems, especially at low attenuation (that is, very little "distance" between client and AP), all the systems show a similar drop-off in performance, maintaining good performance until their signal is attenuated by 90 dB. Although it's possible to map dB attenuation to range in an open-air environment, that's not how radio works inside buildings. Still, these controlled tests provide a fair measure of relative radio performance. Overall, the Atheros radio showed slightly better rate-versus-range performance, followed by the products from Intel and then Broadcom.

We also performed a similar analysis of 802.11g performance in our real-world environment. In this case, performance was averaged across only five data points (different locations), as opposed to the 1-dB increments we measured with the Azimuth system. Unlike the Azimuth tests, this information factors in the effects of the radio module and the notebooks' antenna design. Across all five data points, the average throughput was 11.4 Mbps. "802.11g Real-World Throughput," left, shows the differences from average for all seven systems we tested.

Although average scores provide a concise form of measurement, some of the detail results are notable. In the most distant location, where performance averaged only 0.7 Mbps across all products, for example, Dell-Broadcom delivered the best results, by far, of 1.7 Mbps.

Differences in 802.11a rate versus range were more pronounced than differences for 802.11g. Systems that used the Broadcom radio experienced a rapid drop-off in 802.11a performance. Systems based on Atheros and Centrino radios performed significantly better, with Atheros clearly coming out on top.

These results are depicted differently in "802.11a Azimuth Throughput," at right, which shows the difference from the average throughput (20.99 Mbps) calculated across 1-dB increments from 57 dB to 103 dB. Note that while 802.11g products maintain connections until between 112 dB and 115 dB of attenuation, 802.11a products lose connections somewhere between 97 dB and 103 dB. That, along with the fact that 5-GHz radio signals attenuate more quickly than 2.4-GHz signals, illustrates the notable range differences between 802.11g and 802.11a.

"802.11a Real-World Throughput," page 50, shows results from the real-world office tests of 802.11a. The overall average throughput across all products at all test locations was 12.9 Mbps, but 802.11a connections could only be established at three of the five locations where 802.11g maintained a link. Product differences are slightly more notable than with 802.11g.No Power Shifts

Battery life is always an issue with mobile computing systems, and wireless module vendors have devoted much technology and marketing effort to this area. Intel, for example, heavily promotes the Centrino architecture's enhanced energy efficiency without providing much detail about whether the Centrino wireless module is more efficient than other radio modules.

To address this, we used BAPCO's MobileMark 2005 benchmark to run a series of network battery tests on each platform. Our goal was not to compare the battery life of one vendor's offering to the other, a distinction that has more to do with the laptop battery's size and weight than any differences in network components. Rather, we wanted to assess how each vendor's Centrino-based configuration compared with the non-Centrino offering, when the only variable is the radio module.

The results were unexpected. In every case, using the 802.11 wireless interface provided longer battery life when compared with using the embedded Ethernet interface on a particular platform. With respect to radio modules, the Centrino solution was slightly more battery efficient than the Broadcom-based laptops for HP and Dell. However, in the case of Lenovo, which provided Intel and Atheros radios, the Atheros radio was slightly more efficient (see "Battery Life Test," left).The Software Side of Wireless

Microsoft made Wi-Fi much easier to use when it released its Windows Wireless Zero Configuration system. Like most bundled solutions, however, Microsoft's integrated wireless support provides a lowest-common-denominator solution that may not meet enterprise needs. Microsoft provides support for a limited number of 802.1X EAP types, for example. Enterprises looking to leverage the advanced features defined by CCX (Cisco Compatible Extensions) will have to look to system vendors for support. Finally, there's plenty of room for software innovation aimed at improving user and IT-management experience when dealing with mobile devices that roam among multiple wired and wireless networks.

A common theme among the client utilities provided by notebook vendors is greater flexibility for encryption and authentication, when compared with Microsoft. The inclusion of a multitude of EAP types lets the client authenticate to almost any RADIUS server that supports a common authentication protocol. All vendors included in this test bundle 802.1X supplicants from Meetinghouse Communications. Along with enhanced security, the client utilities provide advanced details on the status of the wireless connection--useful for troubleshooting wireless problems. Information metrics such as SNR (signal-to-noise ratio), data rate, AP MAC (Media Access Control) address, 802.11 channel and client output power level all can be helpful when tracking down connection problems. These utilities also bring support for wireless profiles that change your security and configuration settings based on the wireless network to which you are connected.

Other than Lenovo, which provides its own Access Connection Manager, all the vendors using the Intel 2915 wireless chipset bundled the Intel Proset/Wireless software. The Intel package has a clean interface and still provides adequate advanced detail and troubleshooting options. One notable feature is the auto-import profile option, which lets administrators create one wireless profile and automatically import the settings when placed in the appropriate directory on the client. Adding to the theme of administrative control is the option to password-protect a profile's wireless settings, to prevent inquisitive users from disrupting their own connections. We found it easy to acquire advanced details about the current connection, but were surprised that the exact signal strength could only be found on an advanced statistics screen. The Intel utility's helpdesk features are also first rate. A basic troubleshooting tool, for example, provides notification of connection issues and even includes ways for users to resolve simple problems themselves.

We like Lenovo's decision to deliver a single client utility that works in both Centrino and Atheros notebook offerings. Although most notebook manufacturers bundle the wireless chipset manufacturer's client utility and only make minimal changes, Lenovo wrote its own and provides the same interface regardless of the chipset. The ThinkVantage Access Connections Manager 4.01 provides better connection profiles than similar client utilities. Profiles can be created for wired, wireless LAN, wireless WAN, dial-up and even broadband connections. An office connection can include both the wired and wireless adapter, and when both are connected simultaneously to the same network, the slower interface is disabled automatically. The level of options in each profile is impressive, from enabling the Windows Firewall to changing the Internet Explorer start page automatically when a specific profile is connected. The utility's advanced details aren't as far reaching as Intel's, but the basic connection metrics are included, and the clean interface made it easier to juggle a multitude of connection settings.Dell's Broadcom-based Latitude D410 came with the Dell Wireless LAN Card Utility 3.0, a slightly modified version of Broadcom's client utility. Looking strikingly similar to the Windows Wireless Zero Configuration advanced settings dialog, it includes adequate connection profile settings, but the interface is cluttered. Advanced details on connection are available, including a simple-to-understand signal and noise bar graph listing values discreetly in dBm (decibels below one milliwatt), a feature often buried in other utilities. The site monitor function was particularly useful, with an interface similar to the popular open-source Netstumbler. This utility provides a detailed overview of all APs that can be detected by the client. A basic diagnostics tab makes troubleshooting many standard hardware problems a breeze.

The Broadcom WLAN Utility 3.1 supplied with the HP nc4200 Broadcom-based laptop closely resembled Dell's utility. This software included a wizard for creating profiles that was more intuitive than Dell's, but otherwise, the two utilities are similar.

No matter how well-designed utilities may be, added functions and advanced details can overwhelm some users. The capability to disable the wireless radio in numerous places, with a button or a keystroke, is particularly bewildering. Even advanced users can conclude their wireless is faulty when it's just powered off. Windows Wireless Zero Configuration certainly holds the title for ease of use, but the other utilities are better at connecting to complex networks. The main differentiators are added features that make the job easier for users, administrators and support staff, and the IBM ThinkVantage Access Connections manager best embodies these virtues, with the Intel Proset/Wireless close behind.

Dave Molta is a Network Computing senior technology editor. He is also assistant dean for technology at the School of Information Studies and director of the Center for Emerging Network Technologies at Syracuse University. Write to him at [email protected].

Jameson Blandford is a lab associate at the Center for Emerging Network Technologies at Syracuse University. Write to him at [email protected].Cisco CCX PLays by the rules

Cisco introduced its Cisco Compatible Extensions (CCX) program in early 2003, to provide broader client support for advanced proprietary features of its WLAN infrastructure offerings. Cisco has gone through four versions of CCX in three years. The latest enhancements provide support for Cisco NAC (Network Admission Control), VoIP (voice over IP) performance optimization and call admission control, enhanced power savings modes, enhanced roaming and 802.11 location tag support.

Although the company has been accused of bypassing the standards process to lock customers into an end-to-end solution, Cisco deserves credit for pushing chipset manufacturers to support the broadest possible range of 802.11 and Wi-Fi alliance standards.

Earlier versions have included support for a wide range of 802.11 standard and proprietary features. These include full support for WPA (Wi-Fi Protected Access) and WPA2, including a variety of EAP types. CCX also supports WMM (Wi-Fi Multimedia), a subset of 802.11e QoS; fast roaming using the Cisco CCKM (Cisco Centralized Key Management) protocol; and RF management features such as client RF scanning and reporting, as well as AP-specified maximum client transmission power. Some of these features work with any industry-standard AP, while others work only with Cisco infrastructure.

Conventional PC-oriented wireless NICs typically support the full range of CCX features. Application-specific devices (ASD), such as VoWLAN handsets, are likely to support just a subset of the features. The ASD CCX specification does not require the full range of 802.1X EAP types, for example.Cisco and its CCX partners must maintain a consistent set of features for both newly offered system hardware and the installed base of devices. Although the newest release of a wireless network interface is very likely to support the latest version of CCX, vendors are not required to provide upgrade paths for existing users. Still, we easily upgraded a year-old Dell Latitude D610, from CCX 2.0 to 3.0, by downloading a new executable from Dell's support site.

We have reservations about vendors that abandon industry standards to pursue proprietary approaches, but we see no evidence of this practice in Cisco's CCX program. The company has diligently added support for all newly developed IEEE standards to new versions of CCX and has turned to proprietary extensions only when the features have significant value to customers and equivalent functionality is not available through standard protocols and services.

Executive Summary

Atheros, Broadcom or Centrino? Dell, Hewlett-Packard, Lenovo or Toshiba? These are your main choices when it comes to a Wi-Fi-equipped notebook computer. Select the right combination and the clients will operate seamlessly with your wireless LAN infrastructure. Pick a different client, however, and performance can be affected in unexpected ways.

We tested the Intel Centrino-based clients available on notebooks from the top four vendors, as well as several of those vendors' non-Centrino options. Our tests covered rate versus range (signal strength based on distance from the access point), battery life and software use. Lenovo's IBM ThinkPad T43, equipped with an Atheros AR5211 radio, outperformed the competition in both 802.11a and 802.11g modes. The Atheros-based system also used power slightly more efficiently than the others.By understanding the differences among the major client platforms, you can provide product purchase guidelines to anyone who needs to access your WLAN and optimize infrastructure buildout to the most common clients.

How We Tested

How important is overall system design (hardware and software) to the performance of a wireless network interface on a notebook computer? One of our primary goals in this review was to look at this and other factors--including the design of the radio module (mini-PCI, in this case) and the antenna subsystem--to assess their relative impact. Modern system designers go to great pains to ensure that other system components do not interfere with the radio module.

We began by testing the performance of the system in a controlled and shielded RF testing environment that let us simulate the effects of signal attenuation on performance. We used the Azimuth Systems W-Series WLAN Test Platform to conduct "rate versus range" tests in both 802.11a (5 GHz) and 802.11g (2.4 GHz) modes. The test simulates the performance impact that occurs as a client moves further away from an AP.

This test has two notable limitations. First, since the Azimuth system is directly connected to the mini-PCI radio module, it does not measure gain provided by the antenna subsystem. In addition, since the RF environment is isolated, this test does not account for potentially negative effects of multipath interference, which occurs when radio signals reflect off solid surfaces. Still, the Azimuth system is the industry-standard mechanism for systematically assessing 802.11 radio performance in a controlled environment. It provides us with accurate comparisons between system and radio modules.For our testing, we used a Cisco model 1240 dual-band AP running IOS 12.3(7)JA2. Traffic generation was facilitated by Ixia's Chariot, using a downstream UDP (User Datagram Protocol) traffic script with a data payload of 1,472 bytes. We ran this test continuously, using the Azimuth system to increase signal attenuation gradually, measuring throughput at each step until the link disconnected.

To assess total system performance, including integrated antennas, in a real-world environment, we performed field testing in Hinds Hall at Syracuse University, home to our Syracuse Real-World Labs®. Hinds Hall is a 50,000-square-foot, four-story building constructed in 1955 but fully renovated in 2004. Floors are steel-reinforced concrete, and each floor has 16 structural concrete support columns. All walls consist of modern metal stud and drywall construction. We limited our tests to the two upper floors, which were sufficient for us to evaluate maximum transmission range. These floors consist of walled offices and conference rooms, so we expected transmission range to be less than that of an open, cubicle-style office environment.

All testing was performed between midnight and 6 a.m., after all test and production APs in the building were disabled. We also performed a wireless scan using an AirMagnet Laptop at each test location to verify we were testing in a clean RF environment. We temporarily installed a Cisco 1240 dual-band AP, the same device used in our Azimuth tests, 8 feet above the floor in a location that would be representative of a real-world installation.

After performing some preliminary tests to estimate maximum transmission range, we chose five test locations. One was just outside our lab, approximately 15 feet from the test AP; the other four were in small conference rooms. Two conference rooms were located on the same floor as the AP, at distances of 80 feet (one intervening wall) and 70 feet (two intervening walls). The other two conference rooms were located on the floor above the AP, at distances of approximately 30 feet and 100 feet. In both cases, signals had to pass through a concrete floor.

Each notebook computer was installed on a plastic cart, at desk height (approximately 30 inches above the floor), on a wooden platform affixed to a Sherline Products digitally controlled turntable. The turntable let us test performance using all possible antenna orientations by rotating each notebook computer at a speed of 1 RPM for the duration of the test. The screen angle between the keyboard and the LCD was kept at 120 degrees for the duration of each test, and the starting physical orientation was the same for each system. Testing was performed on 802.11g Channel 6 and 802.11a Channel 64. Chariot was used for traffic generation, using a UDP unidirectional script sending a 1-MB file repeatedly until the test interval was over. In each case, the wireless chipset's power-saving settings were turned off and the laptop was running on AC power.To test battery life, we ran BAPCO MobileMark 2005, which scripts various user actions until the battery is drained completely. The productivity test performs common tasks in Adobe Photoshop, Microsoft Word, Excel and Outlook, thereby keeping the hard drive and processor active. The wireless test scripts Web browsing using the wireless connection to transfer the requested data. The wired test does the same using Ethernet. Whenever the wireless card was not necessary, we disabled it. For every test, we selected Windows' "Always On" profile with the low and critical battery alarms disabled.

All Network Computing product reviews are conducted by current or former IT professionals in our own Real-World Labs®, according to our own test criteria. Vendor involvement is limited to assistance in configuration and troubleshooting. Network Computing schedules reviews based solely on our editorial judgment of reader needs, and we conduct tests and publish results without vendor influence.