Microsoft drops Dryad; puts its big-data bets on Hadoop

Just a month after insisting there was still a place for its own Hadoop competitor, Microsoft officials have decided to discontinue work on LINQ to HPC, codenamed "Dryad."

In a November 11 post on the Windows HPC Team Blog, officials said that Microsoft had provided a minor update to the latest test build of the Dryad code as part of Windows High Performance Computing (HPC) Pack 2008 R2 Service Pack (SP) 3. But they also noted that "this will be the final (Dryad) preview and we do not plan to move forward with a production release."

Dryad was supposed to provide a way for running big-data jobs across clusters of Windows servers. It was designed to provide a platform for developers to build applications that can process large amounts of unstructured data. Just a month ago, Microsoft updated its near-final test build of Dryad.

But it now appears Microsoft is putting all its big-data eggs in the Hadoop framework basket. Microsoft officials said a month ago that Microsoft was working with Hortonworks  to develop both a Windows Azure and a Windows Server distribution of Hadoop. A Community Technology Preview  (CTP) of the Windows Azure version is due out before the end of this calendar year; the Windows Server test build of Hadoop is due some time in 2012.

From the November 11 HPC Team blog post:

"Hadoop has emerged as a great platform for analyzing unstructured data or large volumes of data at low cost, which aligns well with Microsoft’s vision for its Information Platform.  It also has a vibrant community of users and developers eager to innovate on this platform. Microsoft is keen to not only contribute to this vibrant community, but also help its adoption in the Enterprise."

Microsoft Chairman Bill Gates first publicly mentioned Dryad, a Microsoft Research project, in 2006. The company took a number of steps to move Dryad from a research effort to a commercial one.

Microsoft’s stated longer-term goal was to combine LINQ to HPC and its parallel-programming tool stack to create an abstraction layer that would allow users to access compute resources — whether they’re located on multicore PCs, servers and/or the cloud. Microsoft officials said that Dryad/LINQ to HPC would be key to helping the company “turn the cloud into a supercomputer.”

In October this year, Microsoft officials said Microsoft's plan was to continue to work on various alternatives to Java-based Hadoop and MapReduce and that the company was  “still committed” to these efforts. I guess that's no longer the case.

I've asked company officials if there's anything more to say about Dryad. If there is, I will update this post.

Update: Microsoft officials won't say anything beyond what's in the blog post. So no word for those of you asking what Microsoft's plans are re: big data support for those with .Net experience. That was supposedly what differentiated Dryad from Hadoop -- at least according to the Softies a few weeks ago.