World's Fastest Computers Go for Gigabit Ethernet

Despite recent hype about the growing popularity of InfiniBand, the interconnect is struggling to break Gigabit Ethernets dominance of the world’s fastest computers.

InfiniBand accounts for just 120 systems on the latest biannual list of the Top 500 supercomputers released today, compared to 121 in November 2007.

This time last year, InfiniBand appeared to be launching an assault on the high-performance computing (HPC) space, accounting for 127 of the world’s fastest systems, more than treble the number listed in June 2006.

In contrast, Gigabit Ethernet accounts for 285 of the sites on the current Top 500 list, up from 272 six months ago and 207 this time last year.

“It’s pretty clear that InfiniBand is not going to break Gigabit Ethernet’s dominance,” says Bryan Banister, storage manager at the San Diego Supercomputer Center (SDSC), which has one of the largest SANs in academia. “Ethernet is really still there, it’s low latency now, and there is the emergence of Fibre Channel over Ethernet (FCoE).”The exec explains that 10-Gbit/s Ethernet is also helping fend off the challenge of InfiniBand.

”The bandwidth is there with Ethernet - they have 10-Gig now, which is sufficient for most workloads,” he says. “We have a pretty significant 10-Gig data center, and we have 10-Gig links from major research networks such as TeraGrid.”

The Top 500 list, which is compiled by experts from the Lawrence Berkeley National Lab and the University of Tennessee, does not break out specific figures for 10-Gigabit/s Ethernet, although the interconnect features prominently on the world’s fastest system -- the U.S. Department of Energy’s new Roadrunner supercomputer, which as expected, topped the list by a massive margin.

The Roadrunner system, which uses 10-Gbit/s Ethernet to connect 1.5 Pbytes of Panasas storage, offers double the speed of the previous No. 1, IBM's BlueGene/L system in Lawrence Livermore National Lab.

Capable of peak performance of over 1 Petaflop (over 1,000 teraflops, or one thousand trillion calculations per second), Roadrunner is housed in the Los Alamos National Lab and uses 12,960 hybrid "cell engine" processors, which were originally developed for the Sony Playstation 3.In addition to 10-Gbit/s Ethernet, Roadrunner also uses InfiniBand, namely 20-Gbit/s ConnectX adapters from Mellanox and Voltaire switches.

Despite the strength of Gigabit Ethernet, InfiniBand is still a force at the high end of the Top 500 list, according to Jim Kasdorf, director of special projects at the Pittsburgh Supercomputing Center .

”At the high end, InfiniBand is firmly established,” he told Byte and Switch. “The Gigabit Ethernet stuff will be slipping off the low-end – it’s a too low-performance interface.”

More than one third of the top 20 systems, for example, rely on InfiniBand, including the fourth-placed "Ranger" system at the Texas Advanced Computing Center and the Indian EKA supercomputer, which is one of the top non-U.S. sites on the list.

Today's Top 500 list offered some surprises. For the first time, researchers have calculated the energy efficiency of the systems on the Top 500 list, with Roadrunner’s IBM QS22 Cell processor blades offering the top performance of 488 Mflops per second per watt.The list also highlights the growing popularity of quad-core processors, which now account for 283 of the machines on the list, up from just 19 this time last year.

Have a comment on this story? Please click "Discuss" below. If you'd like to contact Byte and Switch's editors directly, send us a message.