Sandia Blasts Off Blade Cluster

The Combustion Research Facility (CRF) at the U.S. Department of Energys Sandia National Labs has replaced a supercomputer in favor of a powerful Linux-based cluster of blade servers for key energy research.

CRF joins the growing ranks of academic and government customers eschewing expensive monolithic systems to build high-powered clusters from standard pieces of hardware (see Luebeck Looks to Clusters and Statoil Builds Dell Cluster).

“It’s fairly well known that commodity computing is the cheapest hardware that you can get,” says Joe Oefelein, senior member of technical staff at the facility. “Our budgets are limited and what this does is maximize what we can get for a fixed dollar amount.”

The CRF, which studies energy conversion for the likes of power plants and industrial furnaces, started building the cluster last August, when it installed 72 blades from Penguin Computing Inc. Each blade contains two Advanced Micro Devices (NYSE: AMD) Opteron processors and runs the Linux operating system.

As well as supporting day-to-day research, the cluster is used to "stage" complex one-off projects before they are deployed on larger DoE supercomputers. These include sophisticated modeling of flame extinction and reignition, says Oefelein. “What we need is the in-house capability to do the day-to-day science, but also stage up to the more ‘grand challenge’ type of calculations."The initial cluster setup offered a peak performance of just over half a Teraflop (trillions of calculations per second), Oefelein says. It wasn't enough. The Livermore, Calif.-based CRF is now doubling that capacity with the addition of 72 more blades. With a new peak performance of 1 Teraflop, the CRF is planning to do some serious number crunching.

Prior to deploying the cluster, the facility relied on an eight-year-old Origin server from Silicon Graphics Inc. (SGI) (NYSE: SGI). Although this fit the bill for awhile, the system had reached “the end of its life,” according to Oefelein, and, like a growing number of other high-performance computing sites, the CRF opted for a cluster (see Buffalo Cluster's a Grid Cornerstone and Panasas Powers Stanford Research).

The new setup is giving CRF more bang for its limited buck -- about $800,000, according to Oefelein, who also cites scaleability and ease of use as major benefits of the cluster. Using a modified version of Linux from Penguin’s subsidiary Scyld Software to actually control the blades helps, too. “Linux is the operating system that we all know,” he says. “Some of the guys here can rebuild the kernels while they are eating their lunch.”

The CRF cluster uses InfiniBand as its interconnect fabric, which speeds up data transfer by “orders of magnitude,” according to Oefelein, although he is unable to put an exact figure on this. Storage comes from nStor Technologies Inc.'s (Amex: NSO) RAID arrays, which provide around 13 Terabytes of disk space.

The cluster would have made it onto the June 2004 list of the Top500 Supercomputing Sites. But things have moved on significantly in the last year, and the lowest-placed site on the most recent list performed at 2.3 Teraflops(see Top Supercomputers Revealed).— James Rogers, Site Editor, Next-Gen Data Center Forum