TOP500 leaders announce new supercomputer benchmark
In the beginning of supercomputer performance measurement there was Linpack. This benchmark, the gold standard for measuring high-performance computing, has been the basis for the TOP500 supercomputer ranking list since 1993. Now, Jack Dongarra, distinguished professor of computer science at the University of Tennessee, Knoxville, creator of the TOP500, and Linpack's inventor thinks the benchmark is showing its age and needs to be replaced.
Why? Because Dongarra believes that Linpack hasn't kept pace with supercomputing needs and must be updated.
"Linpack rankings of computer systems are no longer so strongly correlated to real application performance," Dongarra said in statement. So, Dongarra and his colleague Michael Heroux from Sandia National Laboratories in Albuquerque, NM, are developing a new benchmark that is expected to be released in time for the next TOP500 list release in November 2013.
China passes US on 'World's Fastest Supercomputer' list (photos)
Benchmarks often need updating to reflect changes in how computers are used. The remarkable thing about Linpack is that it's been used for 30-years with little changes. Even the latest version, High Performance Linpack (HPL 2.0), dates back to 2008.
Dongarra explained that Linpack measures the speed and efficiency of linear equation calculations. Over time, applications requiring more complex computations have become more common. These calculations require high bandwidth and low latency, and access data using irregular patterns. Linpack is unable to measure these more complex calculations.
"This is an important issue to address since we are seeing more applications being dominated by differential equations, and thus, each iteration of the TOP500 will show increasing gaps between real versus Linpack performance," said Dongarra.
So he and his colleagues are working on a new benchmark, High Performance Conjugate Gradient (HPCG PDF Link). This new test is being designed to better correlate computation and data access patterns found in many applications today. This could improve the high performance community, because some of today's supercomputers are being designed to gain a high score for the TOP500 list than to meet real-life demands.
"We have reached a point where designing a system for good Linpack performance can actually lead to design choices that are wrong for the real application mix, or add unnecessary components or complexity to the system," said Dongarra.
In their paper describing the new benchmark, Dongarra and Heroux cite the example of the Titan supercomputer at the Oak Ridge National Laboratory. This system has 18,688 nodes, each with a 16-core, 32 GB AMD Opteron processor and a 6GB Nvidia K20 (Graphical Processing Units). "Titan was the top-ranked system in November 2012 using HPL. However, in obtaining the HPL result on Titan, the Opteron processors played only a supporting role in the result. All floating-point computation and all data were resident on the GPUs. In contrast, real applications, when initially ported to Titan, will typically run solely on the CPUs and selectively off-load computations to the GPU for acceleration."
This kind of result, where the benchmark results are misleading about the system's real-world performance is not in the least bit surprising. As long as there have been benchmarks, there have been people playing games with them to artificially inflate their results.
Dongarra hopes that "this new rating system will drive computer system design and implementation in directions that will better impact performance improvement for real applications." He also added that this new benchmark will be adapted to emerging trends, unlike its rather static predecessor.
That said, Dongarra added, "The Linpack benchmark is an incredibly successful metric for the high-performance computing community. The trends it exposes, the focused optimization efforts it inspires and the publicity it brings to our community are very important. Yet the relevance of the Linpack as a proxy for real application performance has become very low, creating a need for an alternative."
Still, Linpack's day isn't done yet. Instead, HPCG will serve as an alternative ranking of the TOP500 list, allowing a re-ranking of the systems on the list to "real" applications. Eventually, the hope is that HPCG will replace Linpack as the supercomputer testing standard.
The next Top500 list, which will have both Linpack and HPCG rankings, will be announced at the Supercomputing Conference, SC2013. This will be held in November in Denver, CO.
Related Stories: