How to Set Up Cisco Nexus Fabric Extender

In a data center that has deployed Cisco Nexus 5000 or 7000 switches, Cisco Nexus 2000 series fabric extenders (FEX) are commonly used for top of rack (ToR) connectivity. FEX units are priced attractively, as they serve a limited purpose and are not feature-rich switches. In fact, a FEX is not a switch, in that a FEX cannot switch traffic locally or be managed independently. A FEX functions only when connected to a Cisco Nexus 5K or 7K series. All traffic flowing into a FEX will be sent down to the parent 5K or 7K for forwarding, even if the destination is on the originating FEX. If you think of a FEX as a remote line card with no local switching capabilities, you've got the idea.

Having established that a FEX is not a switch, let's take a look at the process of installing a new FEX and bringing it online.

1. Rack the FEX. The official Cisco documentation demonstrates racking a FEX in a four-post rack, using the supplied rack ears and rail slides. I have successfully racked FEXs in open two-post racks using only the rack ears. If you choose to do this, be sure to mount the ears further back along the chassis for better weight balance. However you mount the FEX, provide enough clearance for the fiber uplink cables, as they stick out of the front of the chassis. Tight tolerances could prevent the rack door from shutting.

Do not power on the FEX until all cabling and uplink port provisioning on the uplinked 5K or 7K has been completed.

2. Install & cable the uplinks. When uplinking to Nexus 5Ks, one topology option is to dual-home the physical Fabric Extender. Dual-homing a FEX provides path redundancy, but cuts in half the total number of Fabric Extenders you can deploy. A Nexus 5K supports 24 total connected FEX devices, meaning that two 5Ks could support 48 total single-homed physical FEX. When dual-homing, only 24 total FEX are supported between the two 5Ks.

If you choose to single-home the FEX, you lose uplink path redundancy for single-attached hosts. If single-attached hosts are not a concern, a common scenario is to deploy two FEX to ToR, each single-homed to a Nexus 5K or 7K. Multiattached servers then spread their uplinks across the two ToR FEX devices, and in that way enjoy uplink path redundancy. As of this writing, only Nexus 5Ks support dual-homing of FEX.

If you choose to dual-home the FEX, the uplink ports you select must match on both 5K-1 and 5K-2. For example, if you chose Eth1/1 on 5K-1, you must also use Eth1/1 on 5K-2. Another wise design choice is to spread your uplinks over multiple 5500 ASICs (the silicon inside the switch responsible for forwarding traffic). In the Nexus 5500s, ASICs are mapped to groups of 8 consecutive Ethernet ports on the front to the switch, 1-8, 9-16, etc. Therefore, spreading FEX uplinks over ports 1 and 9 is smarter than 1 and 2. If the ASIC servicing ports 1-8 fails, you won't lose both FEX uplinks.

Although options vary by model, FEX can be uplinked with a variety of media. When ordering, be aware of Cisco SKUs that bundle the FEX with uplink media, such as Fabric Extender Transceivers (FETs) or Twinax. Each option has a different price and distance constraint, so research this carefully to be sure you meet your installation’s requirements.

Another consideration is the number of uplinks to use. For example, a 2248TP FEX has 48 10/100/1000 access ports and four 10-Gbps uplink ports. If uplinking all four 10G ports, the oversubscription ratio of access to uplink ports is 1.2:1--very low. At the same time, you’ve used up four expensive 10G ports on the Nexus switch on the other end. Do your traffic patterns warrant using all four uplink ports, or can you get by with just two? Note that from a technical standpoint, the FEX will function correctly with only a single working uplink, but a sensible design uses at least two in a production environment.

The remainder of this example assumes a dual-homed FEX using all four uplinks, connected to a Nexus 5500 pair configured in a virtual port-channel domain.

3. Configure a virtual port-channel and add physical interfaces. Two of the FEX uplinks will homed to one 5K, and two to the other 5K. Then, all four of the FEX uplinks will be combined into a single virtual port channel. Each FEX is assigned a number from 100-199.

The physical interface requires two specific commands to tell the hosting 5K that the interface is servicing a FEX. The command "switchport mode fex-fabric" lets the Nexus switch know that the device on the other end of the link is fabric extender. Note that if you use FETs as uplink media, the switch can’t use these optical modules until this command is in place.

The command "fex associate " tells the Nexus switch which specific FEX is being uplinked to that port. The number selected must match for all uplink ports.

4. Apply the code below to both 5Ks. Once this step is complete, you can optionally prepare your console session to watch the messages that will scroll as the FEX comes up for the first time by typing "term mon".

interface Po101 description UPLINK FEX-01 vpc 101 switchport mode fex-fabric fex associate 101 no shutinterface Ethernet1/1 description UPLINK FEX-01 switchport mode fex-fabric fex associate 101 channel-group 101 no shutinterface Ethernet1/9 description UPLINK FEX-01 switchport mode fex-fabric fex associate 101 channel-group 101 no shut

Next Page: Powering Up5. Power up the FEX. Connect the power cables. Ensure that the redundant power supplies are uplinked to two different power sources.

6. Review console messages. With the FEX powering up, you should see console messages scrolling by on the 5Ks that indicate a FEX has been discovered, ports are coming up (and probably going back down, then back up) and so on.

The first time the FEX boots, it will probably require a firmware upgrade. The Nexus switches to which the FEX is uplinked will upgrade the firmware automatically. The upgrade takes about 10 to 15 minutes. When this process is complete, the FEX will be online and ready for you to document and sanity check.

7. Set the FEX description. This should be simply the name of the FEX. This name shows up in other CLI output, and is helpful in determining just which FEX you're working on.

fex 101 description FEX-01

8. Verify the FEX is uplinked and ready for use. Here are some commands to help you verify the FEX is uplinked correctly:

(click image for larger view) Fabric Extender

9. Provision the FEX access ports. Once the FEX is up and running to your satisfaction, you can provision the access ports according to your procedures. I ensure that all new ports are disabled and described as "AVAILABLE". This makes it easy for operators to see what ports can be used when required, and prevents server admins from plugging into a port at random.

Concluding Thoughts

Note that on dual-homed FEXs, the FEX-related configuration commands on the Nexus switches must be identical. You can do this manually by configuring each switch in turn. Cisco also offers a configuration synchronization service to perform this task automatically, but setting up that service is beyond the scope of this article.

While the output of “show fex detail” shows that all FEX access ports are mapped to the virtual port channel, it is possible to map specific access ports to specific uplink ports using static-pinning.

Finally, Cisco offers a Nexus 5K/2K access layer design guide. I recommend you read it carefully to understand the options and caveats when deploying a Nexus infrastructure.

For more instruction on the Cisco Nexus, see How to Configure Cisco Nexus 5500 Virtual Port Channel

Ethan Banks is CCIE #20655, a host of the Packet Pushers podcast, and IT practitioner since 1995. He can be reached at @ecbanks.