Today's supercomputers: far greater than the sum of their parts

Back in the mid-1990s, I had the opportunity to meet Dr. Irving Wladawsky-Berger, then vice president at IBM, who was charged with building a new generation of supercomputers. But Dr. Wladawsky-Berger didn't go out and spend a huge budget on specialized hardware. Rather, the genius of his approach was to take sets of already-manufactured processors -- at the time, RISC processors from IBM's Unix family -- and lashed them together with a high-speed switch to create supercomputing power. A few years later, the follow-on generation of supercomputers was built in a similar fashion from x86 processors running Linux.

Welcome to innovation, 21st Century style. A couple of months ago, I posted some observations about innovation coming from low-cost and conveniently available parts -- part of the formula for success for our era: “Good enough” may be good enough when it comes to developing new products and services.

One powerful byproduct that has emerged from this paradigm is supercomputers. The latest and fastest behemoth, as a matter of fact, was built from PC chips from Advanced Micro Devices. The supercomputer, as described by The New York Times' Ashlee Vance, runs at the Oak Ridge National Laboratory in Tennessee "and can throw hundreds of thousands of processing engines at computing problems."

"The supercomputing world was long dominated by systems that required specialized chips, memory systems and networking technology. But about 10 years ago, researchers realized they could link thousands of cheaper machines running on mainstream chips and achieve pretty solid performance."

The idea that many small commodity processors could work together to provide supercomputing power is being seen in other forms, such as grids. For example, the SETI@home project employs individual users across the globe, who donate unused cycles to help sift through data in the search for extraterrestrial intelligence.

The Top500 list of the world's current top 500 supercomputers was just released, and just about all are built with commonly available commodity processors. In fact, 79% of the world's fastest supercomputers run on clusters of Intel EM64T processors.

Across all product categories, the high-end brands, typically offered as part of well-crafted and expensive interdependent architectures, will be displaced by more modular approaches offered by commoditizers. In the process, new markets will open as more companies or people have access to more affordable approaches. Or even supercomputers.

This post was originally published on Smartplanet.com