Supercomputer ranking method faces revision

An organizer of the Top500 supercomputer rankings has produced a broader test suite that measures multiple dimensions of a machine's performance. By comparison, a mathematical test called Linpack is currently used to rank systems on the Top500 list, which is released twice a year with much fanfare.

"For a long time it's been clear to all of us (that) we needed to have more than just Linpack," said Jack Dongarra, a University of Tennessee professor who helped create Linpack and who's now working on a suite of tests that go beyond pure number-crunching prowess. "No single number can reflect the overall performance of a machine."

In the world of desktop PCs, where increasing a chip's clock speed by 20 percent rarely yields a 20 percent overall system boost, a similar shift away from simple but potentially misleading measurements has already occurred. Chipmaker AMD in 2002 began discarding the gigahertz labeling system. And this spring, Intel made a similar move for its Pentium and Celeron chips.

The government-sponsored test suite for supercomputers, called theHPC Challenge Benchmark,has pleased some supercomputer makers, such as Cray. But IBM, which is moving aggressively into the supercomputing market and is featured more prominently on the Top500, is more cautious.

The new suite of seven tests won't replace Linpack as the Top500yardstick, Dongarra said. For one thing, the decades-old Linpack permitshistorical comparisons in high-performance computing, or HPC, and foranother, a system that can't get a high Linpack won't do well on othertests, he said.

The new tests grew out of a program the United States governmentlaunched after being spooked by a Japanesesupercomputer called EarthSimulator, which has topped the Top500 since 2002. Theprogram, funded by the Defense Advanced Research Projects Agency (DARPA), has awarded grants to IBM, Cray and Sun Microsystems to develop new supercomputer designs.

"It was done for DARPA and the National Science Foundation and theDepartment of Energy. They wanted something to measure the overalleffectiveness of computers designed for the program, and they realizedthat Linpack was not good enough," Dongarra said.

The next Top500 list is scheduled for release Sunday as the International Supercomputer Conference begins in Heidelberg, Germany.

It's not the first time the benchmark suite idea has been raised. ErichStrohmaier, another Top500 organizer, endorsed a composite test in 2000 to supplement Linpack.

New tests, new fans
Some companies are eagerly promoting the new test suite--in particular,Cray. Five of Cray's X1 systems lead in one test, which measures memorytransfer speed, in contrast to the company's comparatively unflatteringpresence on the Top500 list.

"Customers are always going to want to run their particular codes, butit gives a good understanding about how a system performs in differentareas," said Stephen Sugiyama, a Cray marketing manager. "They've done alot of work to pick a few characteristics about systems that matter tocustomers."

Of the Linpack-based Top500, Sugiyama said, "It's a nice census of veryhigh-performance systems, but when it's used to rank systems, it's notnecessarily a good ranking."

Cray has specialized for years in supercomputing. IBM, though, is tryingto adapt its general-purpose business servers to the market, withsubstantial success with Unix servers and clusters of small Linuxcomputers joined by a high-speed network. And Big Blue is more skeptical.

"Everyone understands Linpack for what it is and what it isn't. No oneunderstands these additional benchmarks in terms of what they are andwhat they are not," said Dave Turek, leader of IBM's "Deep Computing" team. "The hazard isthinking that more benchmarks is more illumination. It might justgenerate more degrees of confusion."

Many tests represent extreme and potentially unusual computingchallenges, and it's not yet clear how well they align with actualcustomer work, Turek said. IBM recommends customers try out theirsoftware before buying a system.

In addition, the benchmark is skewed to reflect the interests ofspecific government agencies, Turek said, alluding to intelligenceorganizations such as the FBI, CIA or the National SecurityAgency.

"Three-letter agencies that all have different kinds of views in termsof what they see as important--they have all stuck something in there toaccommodate their kinds of needs," Turek said.

New dimensions
Linpack measures how fast a system can solve complicated algebraiccalculations--a test that measures processor performance well but notother aspects of a supercomputer. For example, it doesn't address howfast data is transferred to or from memory or disk storage systems.

And though Linpack tests a type of math called "floating-point"calculations, which involve a continuous spectrum of numbers, it doesn'ttest "integer" operations, which involve whole numbers. Integer operationsare used in problems such as processing genetic sequences.

The HPC Challenge Benchmark suite, in contrast, includes tests such asStream, whichmeasures how fast data can be transferred from memory to a processor;Ptrans, which measures how fast one processor in a supercomputer cancommunicate with another; b_eff, which measures the response time anddata capacity of a network; and DGEMM, which multiplies one array ofnumbers, called a matrix, with another.

The benchmark software runs all the tests simultaneously, Dongarra said,so manufacturers won't be able to run just one test or another. However,because the tests measure different aspects of a system, it's notmeaningful to wrap the seven results into a single composite score, headded.

Of the seven tests in the suite, only five are measured, andchanges or additions are possible, according to the Web site.

Meanwhile, the Top500 isn't going away, despite its imperfections.

"It clearly has a place. It does attract a lot of attention in usingthis one number to rate machines. There are some bragging rights that goalong with it," Dongarra said.