Cisco: Oversubscribed by Design

As Cisco Systems Inc. (Nasdaq: CSCO) wades deeper into the SAN market, some of its competitors have been waging a whispering campaign that the MDS 9509 Fibre Channel switch is oversubscribed -- by a ratio of 4 to 1 -- if it's fully loaded with the maximum possible 224 2-Gbit/s ports. When a networking device is oversubscribed, it doesn't provide enough bandwidth for each port to send data at the full rate.

Cisco's response? Of course it's oversubscribed: We built it that way on purpose.

In fact, not only does Cisco own up to the fact that its 32-port line card for the 9509 is oversubscribed -- though it says the ratio is actually more like 3.2 to 1 -- it says the card gives SAN designers more flexibility than competing products.

"The 32-port card is not a compromise in performance," says Paul Dul, product line manager for the MDS 9000 at Cisco. "We've done this purposely because we want to provide differentiated pricing. It's a waste of bandwidth to deploy a full 2-Gbit/s to a host."

For its chassis-based director, Cisco offers a 16-port module, which it says is not oversubscribed and is intended for array connectivity, and the 32-port card for host connectivity. The latter has eight groups of four ports each; each of those groups shares 2.5 Gbit/s of bandwidth to the switch's backplane. Cisco says this architecture reduces the cost per port, with the 32-port line card priced around 60 percent that of the 16-port blade per port."Cisco felt it could compete on price better with the option to use oversubscribed line cards with more ports," notes Nick Allen, analyst with Gartner Inc.

However, Cisco asserts that the 32-port card is nonblocking. "We're oversubscribed, which means you basically have multiple ports vying for a limited amount of bandwidth," Dul says. "Blocking means you have congestion in the network and you're not able to fulfill the utilization of the bandwidth."

The company points to testing it commissioned from Mier Communications Inc. (Miercom) in December 2002. That report said the 32-port card was nonblocking because the throughput was "evenly distributed across all port groups and there were no dropped frames." Using 2,148-byte frames generated by Spirent Communications's SmartBits 6000, Mier measured throughput of roughly 249 MByte/s for each four-port group. The test also checked the 32-port card's oversubscription ratio, putting it at 3.3 to 1.

What do enterprise users think of this whole issue? Barry Brazil, a SAN architect with consulting firm Brazil Hieber & Associates, says the 16- and 32-port modules actually do allow users to design more cost-effective SANs.

"The 32-port blade provides connectivity to initiators and targets that do not need the full pipe, but still uses the same backplane and all feature sets of the MDS," he says. "That is a win/win any way you look at it -- and it still beat the Brocade 12000 in performance."Brazil designed a large-scale SAN for Reliant Energy, a power company based in Houston, which has purchased and deployed several Cisco MDS 9509s and was one of the earliest beta sites for the MDS 9000 switches (see Cisco Beta Site: 'We Love It!').

But Ron Totah, technical marketing manager at Brocade, says that Cisco's oversubscribed 32-port card is suitable only for a customer's lowest-end servers that won't need the full bandwidth of a 2-Gbit/s FC connection. "If you connect high-speed servers to it, you may have to keep some ports unpopulated to sustain performance," he says.

He also points out that the 32-port blade is clearly not intended for handling interswitch links (ISLs): According to Cisco's own documentation, forcing an E-port on the 32-port card disables the other three ports in a four-port group.

"The point is that they're out marketing this as a 224-port switch, but you wouldn't configure it like that in the real world," says Totah.

Cisco concedes that a 224-port MDS 9509 isn't the optimal configuration for a director. However, it says a 192-port system -- with two 16-port line cards and five 32-port modules -- provides excellent overall performance as a core switch.At the same time, Cisco scoffs at Brocade's other cavils, saying the bottleneck in a SAN isn't from the host side (Dul insists that most servers today don't pump data at 2 Gbit/s anyway) but rather, on the array side. For this, Cisco is employing a technology called Virtual Output Queuing, which the company says helps relieve a condition known as head-of-line blocking.

Virtual Output Queuing puts frames waiting for a particular egress port in a separate buffer, instead of having the frames line up one after the other regardless of which egress port they're destined for (and potentially causing congestion). Cisco says each port on its MDS 9509 has 1,024 virtual output queues, which means that in a switch with 256 ports there would be four output queues for each possible destination.

"The most important thing is that you get optimal utilization of array ports, and that's what Virtual Output Queuing provides," Dul says.

As competition grows more brutal in the Fibre Channel switch space, it'll be interesting to see if Cisco will be able to handle other criticisms of its platform with as much effortless (if dul) aplomb.

Todd Spangler, US Editor, Byte and Switch