All-you-can-eat multithreading

Yesterday, I ran across an interesting report by Dan Warne regarding research work at Intel's Shanghai office. It had a number of notable nuggets, such as remote graphics rendering technology that receives wireless video signals from a pocket-sized PC (coming soon to an airline seat near you), or software to assist in data mining of videos and images (which would take ego surfing to a whole new level). Music video production software featured near the end of the article, which was designed to quickly turn your library of home videos into music videos modeled on real ones created by major artists, seemed a bit out of left field, until I remembered this was the Shanghai office, situated as it is in a part of the world that is absolutely Karaoke-crazy. Not that there is anything wrong with that. Feed me enough Soju and I'll be belting out Radiohead or Police songs in the best Karaoke hot spots in Koreatown (albeit in a voice considerably deeper than Thom Yorke's or Sting's).

What most interested this software developing blogger, however, was compiler software designed to turn your single-threaded applications multi-threaded.

Clearly, this is going to be absolutely essential technology, particularly as we bump up against the limits of Moore's law. It's simply not possible anymore to count on single-CPU speed improvements to achieve performance gains given quantum limits related to miniaturization. This means proliferating cores will be the norm. Four to eight cores are quite common now, but in the future, it is likely that cores will number in the thousands.

This creates a lot of problems, because human beings (a group among whom I usually include myself) find parallel programming to be hard. Common problems in multi-threaded applications include:

• A failure to account for the fact that blocks of code running in a separate thread run OUTSIDE all the nice error trapping logic you had in your original flow of programming control, which can cause the entire application to come crashing down because there was an untrapped exception in your process. I did that recently, after quickly tossing a block of code into a ThreadPool thread without putting proper trapping code around it.
• Lack of thread safety: Locks are designed to prevent multiple threads running in parallel from simultaneously accessing shared variables in an application (say, member variables inside of an object). Failure to protect against simultaneous access can cause applications to behave unpredictably, if not crash altogether.
• Too much thread safety: Programmers new to multi-threaded programming tend to quickly grok the need to use locking, but what takes more time is figuring out how much code to protect within a lock. Lock too much, and you have undone all the advantages of a multi-threaded application, resulting in software where all the threads run one after another in a complicated version of single-threaded programming
• Bad choice of code to run in parallel. Even if you lock things properly, if the blocks of code you choose to run in parallel depend somehow on one another, you will have one thread waiting until the other one completes, thus undoing many of the gains from multithreading.
• Deadlocks. A corollary to the previous problem, if you have one block of code waiting on completion of another block of code, which itself must wait on the waiting block of code to finish, your program grinds to a halt. Granted, this is a detectable problem, unless the deadlock only happens in certain situations that are only experienced under heavy load in a production site that, for whatever reason, evaded detection by your stress testing software.

Modern software libraries try to simplify things by creating frameworks that hide the gory details of multi-threaded programming. One such example is the new Windows Workflow Foundation (WWF) framework included as part of .NET 3.0. Visual Studio builds on this base to provide a graphical means to assemble blocks of programming logic into execution trees, leaving the framework to ensure that the tree is executed in an optimized multi-threaded fashion.

Though tools to assist human creation of threadable blocks of code are certainly important, it simply will not be enough to ensure that programs we create can effectively take advantage of thousands of cores. That may seem odd when you consider that the human brain is the most massively parallel computer currently in existence. The fact that I can think of a blog topic, type it out, and process what I see on my computer screen, all the while continuing to breath and digest my breakfast (just to name a few things I am doing right now) is a miracle of parallel programming.

Even so, our flow of conscious thought appears to operate in a mostly linear fashion (I say "mostly," because a few days ago, I woke up with the solution to a software design problem that I had spent days trying to figure out; guess all that hard work had managed to spawn a thread in the deeper recesses of my brain...that, or Rocky the sword-wielding raccoon is a programmer and related the solution to me in my dreams).

Compilers that turn our software creations into multi-threadable programs are thus essential. Programmers will still have a role in guiding the process by defining broad chunks of programming logic through frameworks such as WWF. Breaking that into ever smaller pieces, however, will be a job for compilers.