FCoE: Standards Don't Matter; Vendor Choice Does

In storage area networking, there are really only two vendors to pay attention to: Brocade and Cisco. Combined, these two vendors dominate the Fibre Channel SAN switch market with some 90% of market share by revenue, according to Dell'Oro Group. The rest of the SAN switch vendors just don’t matter, although we'd keep an eye on HP, which is showing more market share (by revenue) of Fibre Channel over Ethernet switches and adapters than Brocade, according to HP, though those numbers come from sales of FlexFabric, which includes Fibre Channel over Ethernet.

Yeah, yeah, revenue isn’t ports shipped, but let's assume the pricing is fairly comparable, and 90% share of anything is still a whopping majority. During the last year or so, some Fibre Channel over Ethernet experts have been arguing nuances of FCoE and which protocols you need or don’t need. FCoE interswich interoperability? I will be old(er) and gray(er) before that happens in any meaningful way. While these are interesting discussions—if you go for that sort of thing—ultimately, they won’t impact your purchasing decisions. You are going to use whatever your SAN vendor tells you to use.

If you are a storage admin, you already know this. If you are not, pay attention. In the storage world, you use the products you are told to use by your value-added reseller (VAR) or storage vendor, and that includes Fibre Channel SAN gear. Go off the approved equipment list, and you go off your support contract. Even though Fibre Channel is an American National Standards Institute/International Committee for Information Technology Standards (ANSI/INCITS) standard, competing vendors' products may not work well together. That there is even some level of conformance in Fibre Channel SANs is due as much to attrition in the product space--two SAN switch vendors and two to three host bus adapter (HBA) vendors—as it does to vendors working together. Or at least that is what Howard Marks, chief scientist and Network Computing contributing editor, tells me.

FCoE (FC-bb-5) is an ANSI/INCITS standard. TRILL is an IETF standard. The DCB protocols are IEEE standards. You don’t think they all get along, do you? Vendors can't even decide which of the many protocols you need, much less agree to get them to work together. One big debate in FCoE minutia is whether there is a need for Quantized Congestion Notification or not. Basically, QCN is a Layer 2 protocol where a node can tell another node to stop sending data when it gets congested.

J. Michel Metz of Cisco wrote a very good blog on QCN and why you don’t need it. You button-down types might like the drier but equally informative Quantized Congestion Notification and Today’s Fibre Channel over Ethernet Networks.

At least, you don’t need QCN when using Cisco equipment because of how Cisco implemented FC-BB-5, where each switch takes the FCoE frame, decapsulates it, processes it using the embedded Fibre Channel forwarder, then encapsulates the Fibre Channel frame into a new FCoE frame and spits it out. You might need QCN with networking gear that hands FCoE forwarding differently, which Metz will tell you if you poke him with a stick a few times.

You don’t need QCN in Brocade's Virtual Cluster Switch (VCS) either, according to Gurpreet Singh, director of data center product management at Brocade. The VDX switches, which form the VCS network, can support QCN, but Brocade has yet to have a request to enable it. Brocades VCS—its multihop FCoE suite of products and architecture—performs Fibre Channel Forwarder functions at the network edge and forwards TRILL frames containing FCoE natively across the spine switches.

Brocade has a decent techincal overview of VCS. To ensure lossless operation, VCS prioritizes control plane frames like neighbor discovery, TRILL control plane and iSCSI control plane at a high priority by default, and any other traffic can be prioritized to be lossless.I know, I know. Both claims bear some further investigation, which I will take up at a later date. Actually, I have a metric ton of questions about FCoE that I want to track down.

What’s that, you ask? How can two vendors implement the same standard in radically different ways? One of them has got to be non-standard, right? Nope. They both appear--to me, at least, based on conversations with both Brocade and Cisco and my reading of the relevant bits of the standard--to conform to FC-BB-5, which is pretty much mum on forwarding mechanisms. FC-BB-5 specifies the requirements that must be met, such as lossless forwarding, preserving order and some other important things, but leaves those pesky implementation details up to the vendors.

I suppose that if Brocade's and Cisco's implementations did conform to the standards, you could make them work together if you wanted to.

Mike Hagen from the University of New Hampshire Interoperability Labs (UNH-IOL) tells me that the next round of testing will be sponsored by the FC Industry Association and will include multivendor interswitch FCoE testing. The question is, why would you want to build a multivendor network (not to be confused with a multivendor DCB network, which is a different animal altogether)? After plunking down the large chunk of change for Brocade's VCS or Cisco's Nexus, learning how to make it all work nicely, integrating it with your management systems and operations, getting your staff trained, and ensuring that all the parts play nicely together, why would you want to add multivendor complexity to the network? You wouldn't. It wouldn’t make sense.

If you are going to converge your storage and data networking, here is what is going to happen. You are going to decide (however your IT department makes these decisions) to purchase whichever product set your storage VAR or array vendor tells you they support. Then you will build it out and make sure it all works well together. You are not going to worry about whether a product does or doesn't use QCN or whether each switch is a Fibre Channel Forwarder. You are going to rely on your VAR or array vendor to do the messy conformance and performance testing for you, and ensure that you can, in fact, ensure lossless FCoE SAN operation of your Ethernet network. You are not going to worry about whether the routing protocol is the IETF's TRILL, IEEE SPB, Cisco’s FabricPath or Brocade's multipath Ethernet routing protocol. (I can’t call what Brocade is doing TRILL. I just can't.)

You're going to use what you are told to use because if you don't, you—yes, I mean you—will take on the operational responsibility of ensuring both your data and storage are going to perform optimally at all times and will work with whatever equipment you acquire in the future. That's a responsibility that I am not confident you can meet, because the products don't have the baseline functionality to achieve seamless interoperability, and there are a lot of unknowns that need to be ironed out.

Nor do I see a compelling reason for competing vendors to work on FCoE feature/function interoperability. If you aren't going to demand interoperability (because everything is customer-driven, so I am told)--and you won’t, because you really don’t want it--the vendors aren't going to do it, and therefore you can't have it even if you did want it. Am I wrong?