10 GIG Can't Wait to Interoperate

In many ways, 10 Gigabit Ethernet is the same as the original 10-Mbps version that the father of Ethernet, Bob Metcalfe, scribbled on a napkin around 25 years ago--only faster. It still has the same header format, the same 8-byte preamble, and the same minimum (64 byte) and maximum (1,518 byte) frame sizes.

Fortunately, there will be no difficult learning curve; it's still pretty much plug and play. How's that for scalability? The biggest change is that CSMA/CD (Carrier Sense Multiple Access/Collision Detection) has been eliminated because 10 Gig will be implemented in full-duplex mode only, meaning that collision detection is turned off. Although some Ethernet purists might consider this a drastic change, it will make your life easier by eliminating the duplexity mismatches that have plagued some Fast Ethernet and Gigabit Ethernet installations. Obviously there will be no shared-media hubs for 10 Gig, but this shouldn't be a big surprise. Practically speaking, shared-media hubs are rarely used today, especially for high-speed versions of Ethernet.

Other major changes involve the interface. The seven types of physical interfaces, or PHYs, are all fiber--there's no IEEE working group focusing on a copper standard. If 10 Gig ever does run on twisted pair, distances would be limited. Each PHY comprises a PCS (Physical Coding Sublayer), which is responsible for controlling the transmitted bit patterns, and a PMD (Physical Media Dependent) layer, which is responsible for converting bits into light signals. The PMD is sometimes referred to as the "optics." These layers were designed to be independent of one another.

With Gigabit Ethernet, you have just two types of standardized fiber interfaces to keep straight: those that support multimode fiber and those that support single-mode fiber. The major difference between single mode and multimode is the light frequencies supported and the corresponding difference in range. Longer wavelengths running on single mode provide more distance.

In contrast, 802.3ae supports three unique light frequencies, represented by corresponding PMDs: 850 nm on multimode, and 1,310 nm and 1,550 nm on single mode (see chart below).The other major change is that there are now LAN and WAN PHYs for each PMD. Multiplying three optics by two PHYs gives six unique interfaces. The seventh interface, sometimes referred to as the LX4 interface, is a LAN PHY and uses light frequencies in the 1,310-nm range. The main difference is that, though the other PMDs convert bits to light in a serial manner, the WWDM (Wide Wavelength Division Multiplexing) interface uses WDM technology to multiplex the bits across four light waves. This interface is the most versatile because it supports both 62.5 micron multimode fiber for short distances (300 meter) and single-mode 9-micron fiber for long-range (10 kilometers) connections.

If you're wondering why there are so many different versions, you're not alone, especially when you consider that there is overlap between LX4 and some of the other standards. A lot of the variation is based on cost, range and the desire to take advantage of existing technologies and installed fiber. For example, the 850-nm optics that drive multimode fiber short distances are less expensive to build than the optics for single-mode fiber going longer distances. The thinking is, Why pay for what you don't need? This makes sense, but distance for the 850-nm PMD is limited to only 26 meters for existing (62.5 micron) fiber. Going 65 meters will require 50-micron fiber, which is much less common.

If you're connecting switches and servers with patch cords in a data center, this isn't a big deal--aside from the burden of keeping track of different fiber types. But for cable installed in structured-wiring plans, it's a different ball of wax. One thing is clear: If you're running fiber today, you should pull some single mode (9 micron) fiber, especially if the runs are more than 300 meters. It will cost you a bit more initially, but the labor cost to add it down the road could be even higher. The newer, higher grades of fiber will increase distances even further.

The WAN PHY makes use of the same PMDs that the LAN PHY uses, so there's no increase in distance. The purpose of the WAN PHY is to make integrating 10 Gigabit Ethernet connections with OC-192 Sonet connections easier. Sonet, the technology used to connect high-speed circuits over the long haul, is reliable but expensive. The WAN version of 10 Gigabit Ethernet is designed to let it interface directly with the DWDM (Dense Wavelength Division Multiplexing) equipment that normally expects connections from Sonet equipment. The other use will be to connect to future OC-768 Sonet equipment supporting OC-192.

The idea is to maintain the low cost of Ethernet while adding enough Sonet functionality to ensure compatibility. This is possible because 10 Gigabit Ethernet's speed is so close to OC-192 Sonet's 9.953 Gbps. The WAN PHY adapts to that speed and adds some Sonet framing, though it does not implement all Sonet's features and is not meant to replace Sonet. For example, 10 Gig does not implement Sonet clocking and cannot participate directly in a Sonet ring. The bottom line, however, is that an ISP, carrier or very large enterprise may be able to use 10 Gig technology to cut costs.Another IEEE working group, 803.17, is developing ways to extend Ethernet's reach across the WAN and directly address the inefficiencies of Sonet. The RPR (Resilient Packet Ring; www.rpralliance.com) protocol is designed to achieve the sub-50-microsecond failover of Sonet rings while maintaining the benefits of packet-based Ethernet, such as low cost, ease of deployment and efficient use of circuits. The group is scheduled to ratify the standard in 2003.

The Metro Ethernet Forum is also working on ways to bring some of the benefits of Ethernet to the MAN and WAN. We hope this will translate to cheaper high-speed WAN and MAN connectivity.

Still Some Flies In the Jell-o

The IEEE 802.3ae standard is all about interoperability. If you use one vendor's equipment on the backbone and another's in the closet, they should work together. The 10 Gigabit Ethernet Alliance has been demonstrating multivendor interoperability at trade shows for the past year, and many vendors have been participating, so there is an excellent chance that products you buy today will be interoperable. Still, it's early enough that you may want to get that in writing from your vendor. Also, some vendors' chassis-based cards will connect to the backplane at only 8 Gbps. This may be tolerable, but make sure you know what you're getting.

As for troubleshooting your 10 Gigabit Ethernet connections, you won't be able to buy a protocol analyzer soon--the vendors we spoke to said they have no plans to introduce these testers. If you have to monitor connections, you'll have to do it on the edges of your network, where there are slower connections. Another option is to use a span port off a gigabit interface on the backbone router or switch. You'll be able to do this if you can limit the 10 Gigabit traffic to less than 1 Gbps while you are monitoring.

People Who Need Bandwidth

So, should you buy 10 Gigabit Ethernet products now? For some technologies--SANs using iSCSI, for example--10 Gigabit's low latency and high speed will be a boon. Certainly, any application that moves around lots of data will benefit. However, the most obvious place to install 10 Gigabit will be the backbone. Look at the utilization of existing Gigabit Ethernet connections using SNMP-based network management or gigabit analyzers. Given 10 Gigabit Ethernet's high entry prices, the last thing you want to do is blindly assume that you need it because you've had some complaints of slow response times. If your utilization is hitting 100 percent spikes--or is heading in that direction--you have a case, though you might be better off trunking Gigabit Ethernet connections together for now and waiting for 802.3ae products to mature and prices to come down. But if you are running out of fiber to accomplish trunking, and leasing or adding more fiber will be very expensive, then the cost is easy to justify.

Most of us don't need 10 gigabits' worth of Ethernet today, but those who do will be greatly relieved. And those who don't need it now will someday. Keep in mind that there will probably be even faster versions of Ethernet in the future. While those who will be working on that goal are probably taking a well deserved rest after completing 802.3ae, speeds of 40 or 100 Gbps have been bandied about. Bob Metcalfe should be proud.

Peter Morrissey is a full-time faculty member of Syracuse University's School of Information Studies, and a contributing editor for Network Computing. Send your comments on this column to him at [email protected].

• 802.17: IEEE working group defining the standard that uses RPR to extend Ethernet over the MAN and WAN.• iSCSI: Emerging IETF standard protocol for extending a SCSI bus over an IP network.

• OC-768: Sonet technology providing 40 Gbps on a single circuit.

• Physical Coding Sublayer (PCS): Part of PHY that dictates the bit patterns sent to the PMD.

• Physical Interface (PHY): Corresponds to Layer 1 of the OSI model; incorporates the PCS and PMD.

• Physical Media Dependent (PMD): PHY layer that dictates the way bits are converted to physical signals, such as light in the case of fiber.Resilient Packet Ring (RPR): Emerging technology combines packet switched networks with dual rings.

• Wavelength Division Multiplexing (WDM): Emerging technology combines packet switched networks with dual rings.

Switch and router vendors Cisco Systems, Enterasys Networks, Extreme Networks and Foundry Networks are shipping or plan to ship this year products that support the 10 Gig LAN PHY. In fact, Enterasys shipped us our first 10 Gigabit Ethernet switch for testing last year (see www.nwc.com/1304/1304sp4.html). Although the company's Matrix E1 Optical Access Switch is not a standards-based product, it certainly proved that 10 Gigabit Ethernet is the real deal. Enterasys says it plans to support all four LAN interfaces by the end of the year and will support the Xenpak transceiver.

The Xenpack transceiver is the 10 Gig equivalent of Gigabit Ethernet's GBIC and should make it easier and less expensive to switch from one PMD to another. But the Xenpak interface is three or four times the size of a GBIC, making it difficult to fit many on one card. There is a new PMD, called Xpak, that will shrink the interface, so that it can use existing board designs. Another initiative, the XFP, promises more dramatic shrinkage, similar to Gigabit Ethernet's SFP (Small-Form-Factor Pluggable), but you won't find it anytime soon because the internal board-level interface component that the interface would plug into has to be developed from scratch.

Size won't matter in the immediate future, though. The vendors supporting these standards on their high-end chassis-based switches offer only one interface per slot, leaving plenty of real estate for multiple Xenpak interfaces. Extreme and Foundry will support the interface, though Cisco indicated no plans to support Xenpak (the company has a proprietary board that makes the optics more interchangeable). Keep in mind that until XFP is implemented, all the interfaces will likely use standard SC connectors.Cisco is shipping cards with 1,310-nm and 1,550-nm PMDs, Extreme supports the 1,310-nm PMD, and Foundry should be shipping all four LAN PMDs by press time. All these will be for the LAN PHY only, with retail prices ranging from $50,000 to $90,000 per port.