SC07 Day 2: Emerging hardware to force drastic changes in programming
"We live in interesting times." That was the message from Tim Mattson, a Principal Engineer at Intel during a tutorial Monday on advanced OpenMP programming. Other presenters at SC07 agreed: software development is much more difficult than it used to be, and it's only going to get worse.
According to Mattson, "This situation resulted from a failure on the hardware side, not a success." It was the inability of hardware to scale up beyond a certain point that lead engineers to rethink the traditional von-Neumann architectures. This lead at first to parallel and multi-core computers, and more recently to special purpose hardware such as GPU (graphical processor unit) accelerators. The future of CPUs will be a heterogeneous mesh of processors with different capabilities, says Mattson. "Energy considerations alone will force us to move in this direction."
Tim predicted that GPUs will ultimately be a dead-end as the functionality of GPUs gets merged and co-mingled with general purpose CPUs. However that's not to say they're a bad idea in the short to medium term. At SC07 dozens of researchers are demonstrating speedups up 20-50x or more for certain algorithms on the GPU. "We're learning a lot from experiments with GPUs," he said, indicating this experience will be invaluable for the future.
Even something as simple as I/O is not straightforward in this brave new world. Consider this graph of disk read performance on an MPI benchmark running on different supercomputers (from the advanced MPI tutorial).
By taking advantage of new MPI-2 parallel I/O functions (level 3 in the slide) this application went saw up to 3 orders of magnitude improvement over posix-style I/O (level 0). It wasn't automatic though. The programmer had to rethink how I/O was done; to think in terms of collective operations done by groups of cooperating processors instead of a bunch of individuals doing their own thing.
With parallel computers, GPUs, FPGA's (programmable hardware), NUMA, and other technologies coming at them from all sides, software engineers face a growing challenge. "Never before in my career have I looked over at the software side and just scratched my head," says Mattson. "I don't know what they're going to do."