What's in (and not) with Microsoft's new high-performance computing service pack

The service Pack (SP) 2 for Windows HPC Server 2008 R2, its high-performance computing product is available for download, as of June 28.

There are a number of new technologies in the latest SP, but there's one key promised component that's not: Microsoft's Dryad.

HPC Server 2008 R2 SP2 adds tighter integration with Windows Azure and Microsoft's high-performance clusters running on-premises. It also provides direct integration between HPC Server 2008 R2 and the Azure programming interfaces, paving the way for "a single set of management tools for both local compute nodes and Windows Azure compute instances," according to a new Windows Azure team blog post.

There's also now a tuned MPI (Message Passing Interface) stack for the Windows Azure network, support for Windows Azure VM role (which is currently in beta), and automatic configuration of the Windows Azure Connect preview to allow Windows Azure based applications to reach back to enterprise file server and license servers via virtual private networks, according to the HPC SP2 post. And there are new features enabling better on-premises clustering in SP2.

But one piece of technology that doesn't seem to have made it into SP2 is the Microsoft Dryad technology (which is going by the official name "LINQ to HPC"). Dryad, which is Microsoft's Hadoop and Google MapReduce competitor, is going to get a second beta "in the coming days," according to the aforementioned blog post.

LINQ to HPC enables developers to write data-intensive apps using Visual Studio and the LINQ programming model and to deploy those apps to clusters running HPC Server 2008 R2. "Programmers can now use thousands of servers, each of them with multiple processors or cores, to process unstructured data and gather insights," explained blog post author Ryan Waite, a Partner Director at Microsoft.

Last I heard, Microsoft's plan was to include Dryad/LINQ to HPC in SP2 and to release the whole thing in the second half of this year. I asked Microsoft execs for additional information as to why Dryad's not in there and whether it has slipped to a later service pack. A spokesperson said Microsoft had nothing to share and no clarifications to make beyond what was in today's blog post.

Microsoft's longer term goal is to combine Dryad and its parallel programming tool stack to create an abstraction layer that will allow users to access compute resources — whether they’re on multicore PCs, servers and/or the cloud. Microsoft officials are relying on Dryad to help the company "turn the cloud into a supercomputer."