I/O, I/O, It's Off to Virtual Work We Go

Part nine in a series. Greg Schulz is the founder of StorageIO and the author of The Green and Virtual Data Center.

Similar to the important role that transmission networks (e.g. power grid) play in the access to and efficient delivery of reliable electrical power, I/O and data networks enable IT services to be delivered to users, leveraging local as well as remote servers and storage. We have previously discussed the role of servers to support the processing needs of information services delivery as well as the associated power, cooling, floor space, and environmental health and safety (PCFE) impacts and issues.

Networks play a crucial role in delivering IT services while enabling a green and virtual data center on a local as well as wide-area basis. It is important to understand the characteristics of various physical and virtualized I/O technologies in order to align those capabilities to different usage scenarios for cost-effective IT service delivery.

There is an old saying in the IT industry that the best I/O, whether local or remote, is an I/O that does not have to occur. I/O is an essential activity for computers of all shapes, sizes, and focus in order to be able to read and write data to memory (including external storage) and to communicate with other computers and networking devices (including Internet services).

Many storage applications are time-sensitive and require high throughput (bandwidth) and low latency with zero data loss. Bandwidth is the measure of how much data can be transferred over a network or I/O interface in a particular time, for example, per second. Effective bandwidth is a measure of how much of the available bandwidth can actually be used, taking into consideration dropped packets and retransmission due to congestion and protocol inefficiency. A common mistake is to look at bandwidth simply in terms of dollars per gigabit per second. The effective or actual usage is important, as is knowing what level of utilization at a given response time (latency level) can be maintained without congestion and packet delay or loss.In general, the faster a processor or server is, the more prone to a performance impact it will be when it has to wait for slower I/O operations. Consequently, faster servers need better-performing I/O connectivity and networks. Better performing means lower latency, more IOPS, and improved bandwidth to meet application profiles and types of operations.

Storage and I/O interconnects have also evolved from various proprietary interfaces and protocols to industry-standard Fibre Channel, InfiniBand, Serial Attached SCSI (SAS), and Serial ATA (SATA), as well as Ethernet-based storage. With the exception of IBM legacy mainframes that utilize count key data (CKD) or extended count key data (ECKD) formats and protocols, open-system computers and networking and storage devices have standardized on the SCSI command set for block I/O. Parallel SCSI cabling still exists but is giving way to SAS, SATA, Fibre Channel, iSCSI, and NAS solutions. The SCSI command set, however, continues to exist.

A plethora of protocols and networks will continue to migrate toward convergence. In the meantime, multiple interfaces and protocols are used to enable tiered access of data and information. Tiered access enables the most applicable tool to be used for the given task, factoring cost, performance, availability, coexistence, and functionality with application service needs. This includes Fibre Channel at different speeds, iSCSI, InfiniBand, NAS, SAS, and other ways to align the access to the level of service needed.

For long-distance scenarios -- such as enabling Fibre Channel or FICON remote mirroring, replication, or backups to support high availability or BC/DR, or clustering requiring low-latency communications -- use FCIP, DWDM, Sonet/SDH, or time-division multiplexing (TDM) MAN and WAN networking solutions and services. For IP-based networks, DWDM, Sonet/SDH, metro Ethernet, and IPoDWDM can be used.

Ethernet is a popular option for general-purpose networking. Moving forward, with extensions to support Fibre Channel over Ethernet with enhanced low-latency and lossless data transmission, Ethernet will eliminate the need to stack storage I/O activity onto IP. IP will remain a good solution for spanning distance or using NAS, or as a low-cost iSCSI block-based access option coexisting on the same Ethernet. Getting Fibre Channel mapped onto a common Ethernet converged or unified network is a compromise among different storage and networking interfaces, commodity networks, experience, skill sets, and performance or deterministic behavior.For the foreseeable future, FCoE will remain for local environments and not for long-distance use. Unlike iSCSI, which maps the SCSI command set onto TCP/IP, or FCIP, which maps Fibre Channel and its ULPs onto TCP/IP for long distance data transmission to enable remote replication or remote backup, FCoE runs native on Ethernet without the need to run on top of TCP/IP for lower latency in a data center environment.

To Next Page

FCoE is in its infancy, however, with a very bright future as the surrounding ecosystems and industry fully evolve and mature. The degree of convergence and the path to get there will depend on timing, preferences, risk aversion, concern over early-generation technology turnover (e.g. hardware obsolesce) budget, and other criteria as well as vendor storage offerings and support. As with other techniques and technologies, the applicable solution should be aligned to meet particular needs and address specific pain points while not introducing additional complexity.

For environments that are not as heavily invested in or committed to Fibre Channel, the opportunity to jump to 10-Gig Ethernet iSCSI or NAS will be appealing for some. For those who do make the commitment to at least one more round of Fibre Channel at 8-Gig, in three to four years it will be time to decide whether to stay with legacy Fibre Channel, assuming 16-Gig FC is ready, or make the jump to FCoE at 10 Gig or the emerging 40 Gig, or jump to iSCSI or NAS or even SAS for smaller environments, as well as a combination of these in a tiered access, or tiered network schema.

Virtual I/O (VIO) and I/O virtualization (IOV) sound similar, but there are distinct differences between them. For example, VIO includes technologies that mask or minimize the impact of performing an I/O operating using RAM or flash-based memory, including solid-state disk devices or caching appliances. The aim of VIO is to provide abstraction and transparency to applications without having to perform and wait for I/O operations to complete. IOV on the other hand, involves emulation and consolidation to improve utilization of I/O adapters and supporting infrastructures, including consolation. Another capability of IOV, also known as converged network solutions, is to simplify and remove the complexity of having to support multiple adapters and associated software.I/O and general-purpose data networks continue to converge to enable simplified management, reduce complexity, and provide increased flexibility of IT resource usage. Converged networks and virtualized I/O are being used at both the server level internally with PCIe enhancements as well as externally with Ethernet, Fibre Channel, and InfiniBand. Even SAS and SATA are a form of convergence by which SATA devices can attach to a SAS controller and coexist with SAS devices to reduce complexity, cabling, and management costs. Examples of converged networks include: Fibre Channel over Ethernet using an enhanced Ethernet; Fibre Channel and Ethernet virtual HBAs and NICs using InfiniBand as a transport; and inside servers using PCIe IOV.

PCI SIG IOV consists of a PCIe bridge attached to a PCI root complex along with an attachment to a separate PCI enclosure. Other components and facilities include address translation service (ATS), single-root IOV (SR-IOV), and multiroot IOV (MR-IOV). ATS enables performance to be optimized between an I/O device and a servers I/O memory management. Single-root, SR-IOV enables multiple guest operating systems to access a single I/O device simultaneously, without having to rely on a hypervisor for a virtual HBA or NIC. The benefit is that physical adapter cards, located in a physically separate enclosure, can be shared within a single physical server without having to incur any potential I/O overhead via virtualization software infrastructure. MR-IOV is the next step, enabling a PCIe or SR-IOV device to be accessed through a shared PCIe fabric across different physically separated servers and PCIe adapter enclosures. The benefit is increased sharing of physical adapters across multiple servers and operating systems.

From a PCFE standpoint, converged networks can be used for consolidation to reduce the total number of adapters and the associated power and cooling. In addition to removing unneeded adapters without loss of functionality, converged networks also free up or allow a reduction in the amount of cabling, which can improve airflow for cooling, resulting in additional energy efficiency.

Wireless networking continues to gain in popularity; however, physical cabling using copper electrical and fiber optic cabling continues to be used. With the increased density of servers, storage, and networking devices, more cabling is being required to fit into a given footprint. To help enable management and configuration of networking and I/O connectivity, networking devices including switches are often integrated or added to server and storage cabinets.

In addition to utilizing cabling that is environmentally friendly, another green aspect of cabling and cable management is to improve air flow to boost cooling efficiency. Unorganized under-floor cabling results in air flow restrictions or blockages requiring HVAC and CRAC systems to work harder, consuming more energy to support cooling activities. Cabling should not block the air flow for perforated tiles on cool aisles or block movement of hot air upward in overhead conveyance systems. Reducing cable congestion has a positive PCFE impact by improving cooling efficiency as well as simplifying management and maintenance tasks.Someday, a truly revolutionary new technology will finally emerge that eliminates the need for I/O operations. For the foreseeable future, however, several things can be done to minimize the impacts of I/O for local and remote networking as well as to simplify connectivity. Keep in mind that the best type of I/O is one that is efficient and as transparent as possible, at least until a new and truly revolutionary way of computing no longer requires a processor to need I/O or data!

Greg Schulz is the founder of StorageIO, an IT industry research and consulting firm. He has worked as a programmer, systems administrator, disaster recovery consultant, and capacity planner for various IT organizations, and also has held positions with industry vendors. He is author of the new book "The Green and Virtual Data Center” (CRC) and of the SNIA-endorsed book "Resilient Storage Networks (Elsevier)".

InformationWeek has published an in-depth report on data center unification. Download the report here (registration required).