Tech Road Map: Keep An Eye On Virtual I/O

Multicore servers allow organizations to consolidate 20 or more production servers into each virtual server host, saving space, power, and staff resources in the process. But feeding these servers enough I/O bandwidth has become a major headache for many data centers.

Some experts recommend Gigabit Ethernet connections for each processor core on the host to ensure sufficient bandwidth, dedicated connections for management and virtual machine migration, plus additional Fibre Channel or Gigabit Ethernet connections for storage, resulting in cable and switch port sprawl.

Leading vendors, including many Fibre Channel providers, are pushing enhanced 10-Gbps Ethernet with Fibre Channel over Ethernet (FCoE) as the solution. Granted, 10-Gbps Ethernet addresses the bandwidth issue, and FCoE converged network adapters can handle the storage side of the equation. However, the standards for both FCoE and Date Center Bridging, the IEEE name for the Ethernet enhancements FCoE requires, haven't yet been ratified.

Other vendors have proposed solutions to the virtual server I/O problem that could be more cost-effective and flexible. These systems create multiple virtual-aware adapters (virtual network interface cards and virtual host bus adapters) assigned to virtual machines. As the VMs migrate from host to host, the virtual NICs and virtual HBAs let them keep their MAC address or World Wide Name, alleviating some network security and zoning issues of VMs.

Virtualization-aware I/O devices' support for quality of service means that virtual NICs and HBAs supporting critical or latency-sensitive applications can use reserved bandwidth and take higher priority than users surfing Facebook while sharing a single physical connection.

To InfiniBand And Beyond

The first alternative to FCoE, championed by InfiniBand vendors Voltaire and Mellanox, connects the servers to InfiniBand switches and uses InfiniBand-to-Ethernet and Fibre Channel bridges to connect to a data center's LAN and SAN resources.

InfiniBand is best known as the high-speed (up to 40 Gbps), lossless, low-latency interconnect at the heart of most high-performance computing clusters. Advocates promote it as the Swiss Army knife of networks, good for remote direct memory access (RDMA) in high-performance computing, IP transport, and even storage connections in applications. On the plus side, InfiniBand solutions bring two to four times the bandwidth of even 10-Gbps Ethernet for server-to-server I/O functions like VM migration.

Voltaire has a 10-Gbps Ethernet/InfiniBand line card that has two 10-Gbps Ethernet ports and 22 10/20-Gbps InfiniBand ports, for its Grid Director 2004 and 2012 switches. The host can use an IP-over-InfiniBand driver and the InfiniBand switch provides Layer 2 bridging and Layer 3 and 4 routing to the Ethernet ports for data networking. For storage access, Voltaire has a storage router with two 10/20 Gbps InfiniBand ports and four 4-Gbps Fibre Channel ports. Hosts use iSCSI or iSER (iSCSI Extensions for RDMA) and the storage router maps iSCSI targets to Fibre Channel logical unit numbers.

At first glance, Mellanox's BridgeX looks very much like the Voltaire approach, with a bridge appliance that has four 40-Gbps InfinBand ports facing the servers and 16 8-Gbps Fibre Channel ports or 12 10-Gbps Ethernet ports facing networks and/or storage. Mellanox's ConnectX cards have dual personalities, so each port can act as a 40-Gbps InfiniBand HBA, like the Voltaire solution, or a 10-Gbps Ethernet port.

The BridgeX is a stateless system where the Mellanox network and storage driver encapsulate Fibre Channel Protocol (FCP) or Ethernet packets inside InfiniBand packets. Where an iSCSI-based system needs to terminate the IP session, unwrap the IP from the SCSI data and rewrap it in FCP, BridgeX just has to peel off the InfiniBand wrapper.

ConnectX is a 10-Gbps card as well, so the server can take advantage of Mellanox's FCoE storage drivers to connect to an FCoE switch like Cisco's Nexus 5000 or Brocade's 8000, or use Mellanox's BridgeX to gateway FCoE traffic to a Fibre Channel switch. When used this way, BridgeX emulates N-Ports to the Fibre Channel switch looking like a series of Fibre Channel HBAs. However, unlike an FCoE switch, BridgeX doesn't provide naming or other fabric services.

While Xsigo uses InfiniBand, its I/O Director isn't an InfiniBand-to-Ethernet and Fibre Channel bridge, but instead uses InfiniBand as the connection media to Gigabit and 10-Gbps Ethernet, and to Fibre Channel modules in the I/O Director. Host systems connect to the 24 20-Gbps InfiniBand ports, but rather than iSER or other InfiniBand drivers, they use Ethernet drivers for the Ethernet ports. Because the Fibre Channel modules use QLogic silicon, QLogic's SANsurfer and Fibre Channel drivers also let admins manage virtual HBAs with SAN management tools. The I/O Director has 15 slots for two-port 4-Gbps Fibre Channel, four-port Gigabit, and 10-Gbps Ethernet modules. Organizations that need higher server-to-I/O densities can use InfiniBand switches between servers and the I/O Director to connect hundreds of servers to a single I/O Director.

Betting against Ethernet has historically been a good way to lose money -- see Token Ring and ATM LANs, for example -- and all three of the vendors using InfiniBand as part of their products are integrating 10-Gbps Ethernet into their portfolios. Mellanox's road map is pretty clear; Voltaire is leveraging its InfiniBand expertise to build high-port-count, low-latency 10-Gbps switches; and Xsigo has been hinting that a future version of I/O Director will use 10-Gbps Ethernet instead of InfiniBand.

Slot Machines

Another group of vendors, lead by NextIO with newcomers Aprius and VirtenSys, promise products that will extend the server's PCIe slots through a switch to an external I/O chassis containing additional PCIe slots. Conceptually much like the PCI SIG's Multiple Root-I/O Virtualization (MR-IOV) standard, but without the need for I/O cards to have MR-IOV support, these systems use a low-cost -- around $200, versus $1,500 for a converged network adapter -- stateless PCIe extender card so servers use the I/O devices drivers unmodified.

PCIe-based solutions can share any Single Root-I/O Virtualization (SR-IOV) compatible card allocating their virtual interfaces to hosts as virtual devices; cards that don't support IOV are assigned to a single host, allowing video, data acquisition, and other specialized cards to be shared across multiple hosts, albeit sequentially.

All three vendors have Serial ATA or SAS/SATA drive bays in their I/O expansion chassis. This lets them create a shared direct-attached storage pool allocating logical drives from an SR-IOV RAID controller in the chassis to hosts, which diskless servers can use for boot or other local storage at a lower cost than a boot from the SAN.

Virtual I/O may have its biggest impact in the blade server market, where a smaller number of I/O channels helps vendors increase server density. Alliances are starting to form, with IBM integrating NextIO technology in its BladeCenter HT and Dell reselling Xsigo's I/O Director.

As a temporary solution until FCoE takes over the world, a lower-cost consolidation point between servers and end-of-row FCoE switches, or a long-term solution, virtual I/O could be worth a look for the more adventurous. On the other hand, those that use new technologies without major-player support always take the risk their virtual I/O system will look like Token Ring in a year or three.