Microsoft closes in on delivering its own Hadoop competitor

Microsoft closes in on delivering its own Hadoop competitor

Summary: Microsoft is delivering a near-final test build of its LINQ to HPC technology, codenamed "Dryad," which competes with Hadoop.


Just days after announcing plans for supporting the Hadoop big-data framework, Microsoft rolled out the near-final version of its own Hadoop (and Google MapReduce) competitor, known as LINQ to HPC (codenamed "Dryad").

Microsoft Chairman Bill Gates first publicly mentioned Dryad, a Microsoft Research project, in 2006. The company has been taking steps, especially recently, to move Dryad from a research effort to a commercial one.

Microsoft’s stated longer-term goal is to combine LINQ to HPC and its parallel-programming tool stack to create an abstraction layer that will allow users to access compute resources -- whether they’re located on multicore PCs, servers and/or the cloud. Microsoft officials have said that LINQ to HPC will be key to helping the company “turn the cloud into a supercomputer.”

(InfoQ cofounder Alex Popescu blogged a comparison of Dryad and Hadoop earlier this year.)

LINQ to HPC was originally expected to be part of the second service pack for Microsoft's HPC Pack 2008 R2-based clustering system. Instead, Microsoft  is integrating the LINQ to HPC runtime and making it a component into Service Pack (SP) 3 for HPC Pack 2008 R2 (which is now at the Release Candidate milestone).

Here's the latest plan for LINQ to HPC, as outlined in an October 17 blog post on the Windows HPC Team Blog describing the coming SP3:

"The HPC Pack service pack is an update to the same Windows HPC cluster software that you know and love, with improvements to basic functionality & stability and a few additional new features such as the integration of the Linq to HPC runtime (previously released as a beta add-on), enhancements to our Windows Azure bursting scenarios by reducing the number of ports you have to open in your firewall (services now use 443 instead of a multiple ones), and the ability to install the HPC Pack software on a server not dedicated to your cluster (e.g. a team file server) for use in a manner similar to the Workstation Node functionality previously available."

HPC Pack 2008 R2 is the operating system for "a cluster of servers that includes a head node, and one or more compute nodes (on-premise of Azure-based)," according to a description on the Microsoft Connect testing site.

The HPC team also announced on October 17 the availability of the release candidate for the Windows Azure Scheduler software development kit (SDK).

Windows Azure Scheduler for Parallel Applications "is a solution that enables you to deploy applications in a scalable, high-performance computing (HPC) infrastructure in Windows Azure," the aforementioned blog post noted. The Scheduler allows users to schedule, submit and monitor high-performance computing (HPC) jobs that use the Message Passing Interface (MPI), service-oriented architecture (SOA), or LINQ to HPC applications, Microsoft officials said.

While on the topics of big data and Azure, here are a couple more related links of potential interest:

  • Microsoft Senior Technical Specialist Buck Woody has a good blog post on the real meaning of big data.
  • Alchemy Solutions has a new version of its NeoKicks product takes mainframe CICS/COBOL/DB2/VSAM-based applications and migrates those mainframes workloads to Windows Azure with what are said to be minimal changes to the application code and data. Microsoft is touting this as potentially aiding companies "that wish to break free of the high cost of mainframes while getting virtually unlimited scale and improved price/performance over their prior environment."

Topics: Microsoft, Operating Systems, Software, Windows


Mary Jo has covered the tech industry for 30 years for a variety of publications and Web sites, and is a frequent guest on radio, TV and podcasts, speaking about all things Microsoft-related. She is the author of Microsoft 2.0: How Microsoft plans to stay relevant in the post-Gates era (John Wiley & Sons, 2008).

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • RE: Microsoft closes in on delivering its own Hadoop competitor

    Too late dude.
    • RE: Microsoft closes in on delivering its own Hadoop competitor

      Its never late...
  • RE: Microsoft closes in on delivering its own Hadoop competitor

    One less reason to buy Yahoo
  • ssss

    <div class="entry">
    Too late dude. </div>
  • 33333

    <li><a href="/reviews">Reviews</a></li>
  • 123123

    <li id="123"><a href="/reviews">Reviews</a></li>
  • RE: Microsoft closes in on delivering its own Hadoop competitor

    <li id="321456">Reviews</li>
  • RE: Microsoft closes in on delivering its own Hadoop competitor

    One thing I don't understand about the Hadoop plans for Windows is how Microsoft plans on integrating .Net languages. Hadoop is heavily optimized for Map-Reduce jobs written in Java, how will I write my jobs in C#?
  • Proprietary Software Always Seems To Be Playing Catch-Up To Free Software

    Why don't they stop copying others, and do something innovative for a change?
  • RE: Microsoft closes in on delivering its own Hadoop competitor

    <a href="" title="mt2" target="_blank">Mt2</a> online 24 saat akdif sunucu tan??t??m sitesi add url ekle add urele free add
    <a href="" title="yemek tarifleri" target="_blank">yemek tarifleri</a> - <a href="" title="Face" target="_blank">Face</a> - <a href="" title="chat" target="_blank">chat</a> - <a href="" title="servar tanitimi" target="_blank">servar tanitimi</a>
    <a href="" title="servar tanitimlari" target="_blank">servar tanitimlari</a> - <a href="" title="server tanitimi" target="_blank">server tanitimi</a> - <a href="" title="server tanitimlari" target="_blank">server tanitimlari</a>
    <a href="" title="sex hikayeleri" target="_blank">sex hikayeleri</a>

    <a href="" title="metin2 indir" target="_blank">metin2 indir</a> - <a href="" title="gm komutlari" target="_blank">gm komutlari</a> - <a href="" title="metin2 gm komutlari" target="_blank">metin2 gm komutlari</a> - <a href="" title="metin2 hile" target="_blank">metin2 hile</a> - <a href="" title="metin2 hileler" target="_blank">metin2 hileler</a> - <a href="" title="metin2 hileleri" target="_blank">metin2 hileleri</a>
  • ygxlpdc 98 zfs

    afojnp,zvdupqcf30, wzqts.