University of Washington

In what could become one of the earliest real-world deployments of InfiniBand, the University of Washington is looking to use the technology as a high-speed fabric to stream multimedia data from its storage infrastructure.

But the slowdown in InfiniBand development has pushed back the university's timeline by a few months, and UW has been waiting for IB vendors to finish the 4x (10 Gbit/s) versions of their products before it can fully implement the technology.

Nate McQueen, systems architect in UW's Advanced Systems Technology Streaming Group, says InfiniBand's combination of low latency and high bandwidth makes it a perfect fit for the gigantic streaming media repository the school has developed, called DigitalWell.

"We're seeing if we can leverage some benefits of InfiniBand's throughput for pulling videos from storage with some of the Fibre Channel-to-InfiniBand bridge technologies," he says.

Currently, UW is using a Gigabit Ethernet switch to connect roughly 1.5 petabytes (that's 1,500 Tbytes!) of data, which is stored mostly in Storage Technology Corp. (StorageTek) (NYSE: STK) tape libraries (see the diagram below).The bottleneck in the system today, according to McQueen, is the TCP/IP stack in Microsoft Corp. (Nasdaq: MSFT) Windows, which runs the system's Microsoft SQL Server databases. "When the servers try to retrieve whole videos, we can only pull 800 streams per server," he says.

To bust through that choke point, UW wants to use InfiniBand. McQueen's group has been investigating the technology for about a year and a half, and it beta-tested the Intel Corp. (Nasdaq: INTC) 1x (2.5 Gbit/s) InfiniBand host channel adapters. Intel has since discontinued its IB silicon products, but the company started an early-adopter InfiniBand program, in which UW is participating (see Intel Plants Wet One on InfiniBand and Intel Bails on InfiniBand).

Now, the university is waiting for 4x InfiniBand gear -- and, more important, Windows software support, which McQueen says third-party vendors have promised by mid-December. UW's streaming group will soon start testing switches from Topspin Communications Inc. and Paceline Systems Corp., he adds.

"With the 4x gear, it will be some time before we get the Windows solution up and running again," McQueen says.

With Topspin's switch, UW will use HCAs based on Mellanox Technologies Ltd. silicon; Paceline's switch will connect via HCAs using IBM Corp. (NYSE: IBM) IB chips."It looks like Mellanox has a lot more driver support," he says. "The stuff from IBM is kind of like, you can run test tools and stuff, but support for SDP [Sockets Direct Protocol] is coming from IBM in the next few weeks."

UW's DigitalWell uses proprietary hierarchical storage management (HSM) software, which the university developed with Sandia National Laboratories. The system can serve on-demand MPEG2 video streams at up to 5.6 Mbit/s over IP, although to access the data at those speeds users must be connected via the high-speed Internet2 network.

Various University of Washington groups and others are using DigitalWell for storing massive amounts of media. For example, UWTV, the university's television channel, hosts 1,500 hours of on-demand video content on the system.

The HSM software provides a way to stream videos from the StorageTek tape libraries with only about a 13-second delay, using direct-attached tape-caching servers. In addition, DigitalWell includes 5 Tbytes of Isilon Systems's NAS storage and 5 Tbytes of RAID arrays from Rorke Data Inc. connected to Brocade Communications Systems Inc. (Nasdaq: BRCD) Fibre Channel switches.

McQueen says UW is beta-testing the Isilon NAS system, which is specifically designed for streaming media, and so far he's impressed with it (see Isilon Is-a-Rich and Isilon Dives Into NAS Pool)."Isilon scales pretty much linearly and still uses one namespace. It does the equivalent of RAID 5 across their boxes... We're trying to get them to look at InfiniBand as a way to do clustering."

DigitalWell's front-end Web servers run Microsoft Internet Information Server (IIS) on six Dell Computer Corp. (Nasdaq: DELL) PowerEdge 1550 1U servers, which are load-balanced using proprietary DNS (Domain Name System) code.

While DigitalWell is currently based on Microsoft's Windows and SQL Server, McQueen says his group has tried to make its programming modules as generic as possible so that they can be ported to IBM's DB/2. "We might move to DB/2 just for scaleability, because of the parallel structure of DB/2. The way you scale Microsoft stuff is by adding a bigger SMP [symmetric multiprocessing] box."

In early 2003, UW plans to make DigitalWell available to all of the university's 25,000 students, offering each 1 Gbyte of storage space for media files. That, McQueen says, will be the real stress-test for the system and the point at which InfiniBand's advantages should start to become even more evident.

"If it's anything like the peer-to-peer stuff, they'll figure out how to use this," he says.Todd Spangler, US Editor, Byte and Switch
http://www.byteandswitch.com