DoD Skeptical About Supercomputer Benchmarks

PITTSBURGH, Pa — The Defense Department is not worshipping false gods as far as supercomputer metrics are concerned. Charles J. Holland Deputy Under Secretary of Defense for Science and Technology, said at SC2004 here that paying attention to the Top500 list of the world's fastest supercomputers is misleading for evaluating high-performance computing platforms for DoD.

"High-end computing enables the U.S. to solve 'unsolvable' problems first," said Holland. But 'faster' hardware, in isolation of the application and business context does not necessarily equate to added value".

The DoD applies the High Performance Linpack (HPL) metrics instead to make supercomputer buying decisions, said Holland. HPL not only measures speed but also accuracy of calculations within limited performance parameters.

Compounding the problem of choosing new machines is that progress in software has been moving at "glacial speed" compared to hardware in high performance computers (HPC), said Holland in an invited talk. "One challenge is end-to-end latency over time: it has not been so good for the last three years. Another is that much of written code cannot utilize an entire HPC effectively."

Holland noted also that code does not scale with processors in various architectures. In order to confront the software HPC problem Holland is applying a strategy he has used over 20 years at his earlier position at the National Security Agency: observe, orient, decide, and act mentality to find a faster path to solutions.DoD software is growing in size and complexity to where by 2010 close to 20 million lines of code will be used to control avionics in a typical fighter aircraft. "We need to apply a systems engineering approach to both practical problems and to our education system," said Holland. "We now have a multi-agency effort to use common analytical methods and tools to map different applications to appropriate hardware and software architectures."

Holland said that a new set of benchmarks called HPCchallenge have been introduced that augment HPL benchmarks that more closely can be applied to real-world applications. "It's a way to understand what happens at the edges of DoD applications."

At SC2004, a Friday (Nov. 12) panel will describe the HPCchallenge benchmark framework. The panel will examine the architecture stresses of the benchmarks, the relationships between these benchmarks and real application performance, and the political and business motivations to develop and publicize the HPC challenge benchmarks that look beyond High Performance Linpack (HPL) and the Top500 list.