Will There Ever Be A Right Time For IO Virtualization?

Last year, Xsigo and Dell announced a relationship where the server vendor would resell the IOV supplier's equipment. The move was expected to position Dell at the forefront of virtualization. However, difficulties in integrating IOV into servers cleanly and advances in bus and IO performance have done little to advance IOV outside of relatively niche situations like high performance computing, data modeling and animation rendering.

"As our customers continue down their path of IT simplification, they look to Dell to help them with all aspects of their virtual infrastructure," said Rick Becker, vice president of Software and Solutions, Dell Product Group. IO virtualization was Dell's next step down the virtualization route, as a valuable offering for their customers. 

Vendors rushed into fill the market. Xsigo was followed by Virtensys, NextIO, Neterion and Aprius. Regardless of the player, the technology shared some commonalities. All IOV technologies are based loosely on the underlying principle that each physical server has one or two cable connections to a rack mounted card cage, into which all the I/O resources like networking and storage are plugged. Each virtual machine (VM) is subsequently allocated a dedicated virtual NIC and virtual HBA, giving them dedicated I/O access. This process allows the VMs to improve performance by bypassing the hypervisor when attempting to gain access to IO resources.

Vendors vary in their choice of protocols to the card cage, some preferring PCIe while others selecting InfiniBand or Ethernet. They also differed in features of the card-cage and the number of available interconnections. Until now, the various IOV vendors have been ironing out their solutions and placing their bets on the right architecture, as the data center market is not yet in need of this technology.

A year later, Dell has become a little less positive about its Xsigo deal. "Xsigo has advantages and disadvantages," says John VonVoros, virtualization product manager at Dell. "They're a little nichey and require a lot of additional work [to support the adapter drivers and the Infiniband infrastructure]." What happened between Dell and Xsigo underscores some of challenges faced by IOV vendors in general. The technology that burst onto the scene at the end of 2008 was supposed to extend the virtualization wave that took over the network and server to the I/O. IOV was expected to enable interconnectivity redundancy and drastically reduce the network card and cabling infrastructure of the data center and supposedly lead significant power savings.IOV was particularly important, the theory went, as multi-core servers are introduced with increasingly powerful processors, I/O was destined to be the limiting factor in virtualization. As such IOV would deliver performance gains enabling servers to run far more virtual machines more efficiently. "Single Root IOV unburdens the CPU by offloading the cycles that would otherwise be dedicated to the software switches inside of hypervisors," says VanVoros. "This is important for two reasons. First, we can get more VMs on a single server because I have more CPU bandwidth available. Second, for the applications that are virtualized, I may be paying a CPU-based licensing fee on the software, which makes any 'stolen cycles' expensive. Oracle charges for licenses by the core for their database. Those CPU cycles are very expensive," says VanVoros.

Some quick math shows that IOV results in more efficient use of CPU resources. "We did a technology demo with Intel at Intel's Developer Forum last year that showed a 7 to 13 percent reduction in CPU loading on VMware wHen using SR-IOV. The amount of savings depends on how much you are asking the software switch to do, such AS vLAN administration, packet filtering, etc. Our math shows ~5 more VMs per server or you could reserve this for peak demand. So, IOV came in to allow vast numbers of VMs to reside on a 1U physical server; the cost savings were clear from day one," says VanVoros.

In fact, virtualization adoption is still in its infancy making the need for IOV nominal. "Only 16 percent of enterprise workloads are running on virtual machines today" says Gartner analyst Tom Bittman. I/O Virtualization only starts to make sense when 20or 30 VMs are on each server, says VanVoros, then the software switch in the hypervisor consumes a lot of resources. If you have three or four VMs per server, IOV is not all that important. On average, he says that he sees five to eight VMs per server.

Part of the reason for virtualization's extended incubation period has to do with the fact that if you need to reduce server costs, the thing to focus on is application licensing not server hardware. "When you look at licensing in general, the fact that you spent $8,000 on a server can be 10 to 15 percent of the overall server cost," VanVoros says, "So if you're trying to save money by virtualizing applications, you may just dedicate a server for that one application and then have pool of servers for virtualized applications." While servers are not overly expensive and the price is only going down, consolidating them is not going to slash data center costs the way things are working at the moment.

Today, the market for IOV remains largely in high-end computing and biggest virtualization shops. In those markets, engineers are eager to push their hardware with maximum virtualization density and need IOV to support so many VMs, precisely the market where Xsigo fares well. One of Xsigo's first customers was Avaya, who use Xsigo's technology for their developers, to run diverse application environments for R&D scenarios. Using IOV, Avaya has enabled them to test applications without the need to give each developer a number of physical server to play with. Xsigo has built their infrastructure on such high-powered computing by using InfiniBand to promise low latency high bandwidth communication.The larger market for IOV will be the standard enterprise data center this time next year, where performance will become less of an issue than compatibility, reliability and flexibility. "I would expect in the next 12-18 months that IOV would become part of any server configuration," says Von Voros.

Too move into the server, and ultimately onto the motherboard, IOV will need to find broad based adoption within the data center. This implies that a common, ubiquitous protocol will be used to base wide spread adoption for IOV. It is extremely unlikely that data center engineers will be interested in moving to InfiniBand throughout the data center for the sake of this technology. More likely IOV will succeed in the mainstream based on common standards like Ethernet and PCIe using the Single Root IOV system as laid out by the PCI special interest group, which will enable far smoother integration into the data center.

Aprius claims to meet those criteria, by supporting any standard PCIe card and supporting the card vendor's drivers as well as being based on Ethernet with 10GbE, used to connect their 'gateway' to the servers. The move positions Aprius to partner with Ethernet switch and infrastructure vendors who would normally reject an IOV solution based on Infiniband or required proprietary technology. Yet Aprius too faces its own challenges. The company has yet to ship working product while their competitors have been shipping equipment for some time. Even some of Aprius's major benefits are starting to change. In April, Xsigo announced a software change that allows its drivers to work with any card.

Although right now IOV may not appearing on the shopping list of most data center architects, that could change as server virtualization grows and the value of the technology becomes more apparent.