Analysis: Carrier Ethernet

Are frame relay's and sonet's days as kings of the WAN-connectivity hill coming to an end? The evidence is there: Carrier Ethernet port shipments are projected to skyrocket, with growth of more than 700 percent between 2005 and 2009, according to Infonetics Research. The analyst firm further estimates that worldwide sales of Carrier Ethernet switches and routers will hit $4.6 billion by 2009. As for the "whys" of this shift, our reader poll revealed that IT pros like Carrier Ethernet for LAN-to-LAN networking, Internet access, VoIP, file transfers and business continuity/disaster recovery. All these deployments require high data rates and low latency, and Carrier Ethernet, aka Metro Ethernet or Ethernet Services, beats other WAN technologies cold when it comes to delivering improved reliability plus more--and more flexible--bandwidth, generally with lower costs.

We say "generally" because last-mile connectivity is still inducing some sticker shock. Yes, service providers claim that cost savings of 20 percent to 50 percent can be gained by choosing Carrier Ethernet over comparable WAN services, and this is largely true ... when service is within a single metro area, with existing fiber, and when providers can use lower-cost switches and pass that savings along to customers.

However, prices for long-haul connectivity haven't dropped as dramatically. Providers we spoke with point out that last-mile connectivity is still a significant cost, especially when facilities need to be brought in to support Carrier Ethernet at the point of presence in addition to laying fiber to the building.The IEEE 802.3ah EFM (Ethernet in the First Mile) standard may help with last-mile costs, but it has limitations. And Carrier Ethernet is still generally available only in larger metro areas.

What's Here Now?

To get a feel for current Carrier Ethernet offerings we issued an RFI on behalf of our fictional company, Tac-Doh, which has eight sites within the continental United States interconnected over frame relay, DSL and leased lines. Three data centers are meshed with OC-3, with remote sites linking to the data centers using T3 frame relay or IPsec VPNs over DSL (see "Current WAN Setup" in the gallery).

We asked several service providers to fill out our detailed Carrier Ethernet RFI. Most politely declined--carriers are notoriously squirrelly about divulging information, especially on pricing, and divulging information is the name of the game in RFIs. Only Qwest Communications had the courage of its convictions and returned a response. Its service offering addressed our concerns, and its plans for expansion mesh well with what enterprises have been telling us is important. Our original RFI and Qwest's response are downloadable (at right).

After considering our bandwidth requirements and the differing CoS (class of service) requirements for our critical applications, Qwest engineered a Carrier Ethernet link with minimum and maximum bandwidth settings (see "Bandwidth by Location" ). Combined with our SLA goals, Qwest's proposed Carrier Ethernet offering seems to fit our needs. It doesn't come cheap, mind you, but the $142,000 monthly charge for a 36-month gold package contract does include WAN costs, a Cisco Catalyst 3750 at each location, and 24/7/365 management and monitoring.As mentioned, last-mile connectivity is a big part of the still-high cost of Carrier Ethernet. To offset the hit to enterprises, Qwest told us it might eat some of the expenses if it can resell services to other customers locally, or initial costs maybe rolled into the monthly charge. Paying for last-mile installation over 36 months may be more palatable than forking over big money up-front.

When building RFIs for multiple providers, structure cost charts so you can make apples-to-apples comparisons. Insist on a detailed breakdown to ensure you know exactly what you're paying for, both in upfront and monthly recurring charges.

Building Blocks

One of Carrier Ethernet's charms is the familiar technology behind it. Rather than having a digital line terminated at an edge router or frame relay access device at the customer location, handoff is 802.3 Ethernet, allowing for support of common and critical protocols like 802.1Q VLAN and 802.1p prioritization.

A Carrier Ethernet network can be treated like a switch: packets enter and exit as Ethernet frames. Within a single metro area, frames are switched through the cloud. For long-haul links between cities, frames are encapsulated in a VPLS (Virtual Private LAN Service) using Multiprotocol Label Switching, L2TPv3 or GRE to create the network.The benefit of MPLS--which is what Qwest's iQ Networking service relies on--is that by sharing long-haul links with other traffic, costs drop compared with a dedicated Sonet circuit. Qwest also offers Ethernet over Sonet, which forms a private circuit with the network connection dedicated just to us--but that service comes at a much higher price. Its recommendation, and we agree, is for Tac-Doh to use VPLS To find out how prices for long-haul Ethernet services compare with SDH/Sonet, see our exclusive cost analysis by TeleGeography's Greg Bryan.

Our network options at that point are to treat the WAN as a switch, directing LAN-to-LAN traffic using VLANs, or to keep our routers at the LAN/WAN edge, but under our administrative control. However, migrating from a routed WAN--which is a typical arrangement--to a switched WAN will require significant changes to our underlying network topology, and significant engineering to flatten the network.

The lesson: While Tac-Doh can still reap Carrier Ethernet's lower cost-per-bit benefits, green-field WAN installations will benefit the most from Carrier Ethernet. Architecture decisions will definitely be easier.

One saving grace is that with Carrier Ethernet, Tac-Doh can migrate from routing to switching slowly, by provisioning one VLAN for routed traffic and adding VLANs for switched traffic. From Qwest's perspective, a VLAN frame is a VLAN frame, regardless of origin or destination.

Take The E TrainsCarrier Ethernet is generally sold in one of two configurations: E-Line is a point-to-point Ethernet link, while E-LAN provides a multipoint mesh. In both cases, connections are logically terminated using a UNI (Universal Network Interface). Multiple UNIs can be attached to one physical termination point, allowing a single WAN link to support multiple Carrier Ethernet services simultaneously.

E-Line is ideal in situations were high speed, low latency and data privacy are high priorities, making it a best bet for data-center-to-data-center connections and disaster recovery. E-Line is similar to frame relay in that each E-Line connection forms a Ethernet Virtual Circuit over the shared Ethernet network.

E-LAN is where the power of Carrier Ethernet really takes off. Unlike conventional WAN technologies like TDM, frame relay or Sonet, E-LAN Carrier Ethernet turns the WAN into a mesh. An E-LAN can simply switch packets where needed, or 802.1Q tags can direct frames to specific locations, just as on the LAN. The carrier is responsible for delivering frames to their respective destinations.

An interesting side benefit of E-LAN is that you can flatten your network topology and do away with edge routers altogether, opting to switch traffic from place to place. If your WAN topology already includes edge routers--and whose doesn't--you can continue to use them, but doing so defeats one of the design benefits of Carrier Ethernet. However, flattening does require that the carrier intelligently handle broadcast traffic, lest it overwhelm your network.

Location, Location, LocationThe $64,000 question is: "Can I get in on this deal?" Not surprisingly, the answer is, "It depends." Like most new telecommunications services, high-value, high-population centers are serviced first, with supply trickling into smaller markets later. Carriers like BellSouth, Qwest and Verizon are pushing down-market for metro services, but nationwide offerings are just now being announced. Our expectation is that once Carrier Ethernet has a bigger footprint, the cost of services will drop, and switching from frame relay or an MPLS VPN will be a no-brainer.

So, what's the holdup? In large part, the lack of fiber to buildings. Don't just assume that your site in Boston, Los Angeles, New York or San Francisco will have capacity for Carrier Ethernet. And fiber coverage in smaller markets is even slimmer. A recent survey of carriers by Heavy Reading indicated that new fiber build-outs and wire-line access platforms, such as Ethernet over copper and Ethernet over TDM circuits, are critical for deploying Carrier Ethernet to subscribers (see "Factors Influencing the Extension of Ethernet).

Open Up And Say Ah

The IEEE 802.3ah EFM (Ethernet in the First Mile) standard may be the answer to the dearth of last-mile fiber from the central office (CO) to customer site, because it standardizes Ethernet connectivity over existing voice-grade copper cabling--by far the most common first-mile facility. However, there are some significant limitations to 802.3ah's real-world usefulness. The first, and foremost, is bandwidth. TDM circuits are generally slower than LAN speeds, and increasing WAN throughput often means adding more TDM circuits or jumping to a higher-speed link, say from T1 to T3.

Distance restrictions also hamper the reach of Ethernet over copper. The 802.3ah limitations for native Ethernet over voice-grade copper, for example, are 10 Mbps over 750 meters cable length, 5.7 Mbps over 3,000 meters.Qwest's service offering in our RFI assumed fiber to Tac-Doh's buildings--but include installation if needed--and would use Qwest's native Ethernet Local Access service. Qwest's proposal places a managed Cisco 3750 switch in each building, with fiber links to local COs. From there, each local CO would provide a fiber connection to the nearest Qwest network.

Where fiber is unavailable, Qwest could provision Ethernet over ATM (RFC 2684 Multiprotocol Encapsulation over ATM Adaptation Layer 5) or over frame relay (RFC 2427 Multiprotocol Interconnect over Frame Relay). Since Tac-Doh is a figment of our imagination, of course it has fiber to each location. Companies living in the real world may have to either invest in infrastructure upgrades or wait awhile for the benefits of Carrier Ethernet, but it's a connectivity option that bears watching.

PER-MEGABIT PREMIUM: How do prices for long-haul Ethernet services compare with SDH/Sonet? See our exclusive cost analysis by TeleGeography's Greg Bryan.

RFI Scenario: Tac-Doh

TAC-DOH'S CLASSIC hub-and-spoke frame relay network has served its needs well, but now, like many organizations, the company is asking more of its WAN. It's ready to undertake a complete network redesign and is interested in the flexibility and much-heralded cost savings of Carrier Ethernet.Tac-Doh runs many delay-sensitive applications across the WAN, including VoIP for internal communications, video, as well as SAP and database transactions. Internet access and general bulk traffic, like file sharing, round out the mix. We weren't interested in remote or SOHO access in this RFP.

We needed a bandwidth boost, and speed, latency and delay are all critical factors to consider. While Qwest couldn't provide end-to-end guarantees for our RFP without engineering the whole enchilada--including contracting with local exchange carriers for the local loop--it did provide us with targeted goals (see "SLA Goals").

Latency and jitter SLAs are well within specs for time-sensitive applications like VoIP and video. By applying CoS (class of service) to prioritize packets entering the Carrier Ethernet network, Tac-Doh should be able to sustain simultaneous voice and data.

Tagged frames are mapped into one of four CoS categories. The highest priority is used for critical network protocols like routing, while other traffic would primarily fall into the three lower classes. Prioritization is used when the local loop is at capacity, and for prioritizing traffic across the WAN. Essentially, Qwest wants to make networking as transparent as possible, leaving traffic engineering decisions in the customer's hands. During initial rollout, Qwest engineers will work with Tac-Doh to map traffic requirements to the four classes of service. Post-rollout changes are managed through the Qwest Control management portal and are usually implemented less than 24 hours after the change request has been reviewed and authenticated.

Poll PositionsWe selected locations in major hubs like Chicago, San Francisco and New York, as well as smaller urban areas such as Waltham, Mass.; Green Bay, Wis.; and Syracuse and Manhasset, N.Y. That mix is because large businesses aren't limited to big metro areas: Syracuse, for example, counts Carrier Corp. and Lockheed Martin as two of its main employers, while a number of smaller, geographically dispersed companies are setting up shop.

Along these lines, in our reader poll we asked about the geographic dispersal of readers' WANs; interestingly, of those with plans to deploy Carrier Ethernet, most were within a single metro area (see "How Many Locations Do You Connect or Plan to Connect?").

Multiple metro areas in a region and across the country combined to made up the bulk of the remaining responses, while international Carrier Ethernet was virtually nonexistent, at 9.4 percent. Not surprisingly, when the number of sites increases, the geographic spread of the Carrier Ethernet WAN grows as well (see "What Is the Geographic Reach of Your Current or Proposed Carrier Ethernet Sites").

Ethernet: Long-haul Premium, Local Savings

Carriers have been talking up long-haul Ethernet rollouts for some time now, but service has been slower to materialize than expected. Given limited availability, it's too early to predict how prices will move compared with similar SDH/Sonet circuits, but in the near term it seems certain that Ethernet prices in the United States and Europe will represent a premium above SDH/Sonet service in the long-haul.

Yes, this goes against the conventional wisdom that Ethernet is the least expensive mode of transport. This is because when long-haul Ethernet is delivered over SDH or Sonet, you have to account for the base bandwidth cost, plus a premium. It's in end-to-end service where customers see savings--lower equipment costs, inexpensive Metro Ethernet services and reduced costs for incremental upgrades make Ethernet a value play.TeleGeography Research has tracked wholesale bandwidth prices for capacities ranging from T-1/E-1 to 10-Gbps wavelengths for several years. Data points represent wholesale, POP-to-POP SDH/Sonet and DWDM circuits, exclusive of local access.

In 2006, TeleGeography added survey data for Fast Ethernet (100 Mbps) and Gigabit Ethernet (1000 Mbps) Ethernet over SDH/Sonet services, based on the growing number of carriers deploying long-haul Ethernet. Prices are for full, protected circuits and are collected as monthly recurring lease charges in U.S. dollars. Install fees do not apply, and the price per megabit is calculated from the carrier-provided MRC (monthly recurring charge).

We saw numerous carrier long-haul Ethernet roll-outs in 2006, though the service is still far from ubiquitous. For example, in a midsize U.S. city such as Denver, long-haul Sonet customers can choose from more than a dozen providers but, according to TeleGeography's most recent surveys, Denver has fewer than 10 providers offering long-haul Ethernet. Cities on the coasts generally see 20 to 30 service providers offering Sonet, compared with 10 to 15 long-haul Ethernet providers. Only New York boasts more than 20 Ethernet competitors, with 36 companies providing long-haul Sonet.

In Europe, TeleGeography has seen somewhat similar trends, though most European cities have more service providers selling long-haul Ethernet service than their North American counterparts. Thirty-one carriers offer long-haul Ethernet in London, with 40 providers selling long-haul SDH. Competition is similar for larger cities like Paris and Frankfurt, Germany. Madrid, Spain; Milan, Italy; Stockholm, Sweden; and other smaller metro areas are closer to U.S. levels, with about 10 to 15 Ethernet providers as compared to 20 to 25 SDH providers.

Based on 2006 pricing data and conversations with pricing managers at major global and regional carriers, TeleGeography has determined that POP-to-POP long-haul Ethernet circuits are priced at a premium compared with SDH/Sonet. On most intra-U.S. routes TeleGeography surveys, wholesale customers pay approximately 15 percent to 30 percent more per megabit for GigE, compared with an OC-12 (622 Mbps) link. The price bump is somewhat higher--sometimes dramatically so--when comparing FastE to STM-1/OC-3 (155 Mbps) on intra-Europe and trans-Atlantic routes.Ethernet pricing structures are best understood by comparing the dollars per megabit for an MRC to long-haul Sonet circuits of a similar full-port capacity. On a common long-distance route, say Los Angeles to New York, per-megabit pricing for GigE service ranges from $31.99 to $41.40. The per-megabit price range for OC-12 services on the same route, looking at carriers that also offer GigE service, is $20.72 to $28.43. GigE prices per megabit are, on average, about 25 percent higher than OC-12 prices.

Further investigation on other routes demonstrates similar trends. One major bandwidth provider offers a GigE circuit on the New York-to-Washington route for $9.96 per megabit, compared with $8.20 for an OC-12. This prices GigE about 21 percent higher than a traditional 622-Mbps circuit. For Chicago to New York, GigE prices range from $16 to $27 per megabit; OC-12 MRCs from carriers offering GigE on the same route range from $13.75 to $22.53, averaging about an 18 percent premium. For connections between most U.S. business centers, the dollar-per-megabit markup between OC-12 and GigE is 15 percent to 20 percent.

Long-haul Ethernet in Europe is typically sold at a very high premium compared with SDH service. For this dollar-per-megabit analysis, FastE prices are compared with STM-1 (155 Mbps) SDH circuit pricing. London-to-Paris, POP-to-POP FastE prices range from $23.10 to $28.62, while the STM-1 dollar-per-megabit price range is $11.22 to $15.59 for the major pan-European carriers surveyed. This pricing structure is similar with trans-Atlantic routes--the dollar-per-megabit price for a FastE circuit from London to New York ranges from $35 to $46.20. STM-1 prices for that route for carriers also offering FastE range from $19.35 to $34.50. --Greg Bryan, TeleGeography research analyst