Microsoft drops Dryad; puts its big-data bets on Hadoop

Microsoft drops Dryad; puts its big-data bets on Hadoop

Summary: Microsoft is dropping its 'Dryad' big-data processing work and focusing, instead, on developing a Windows Azure and Windows Server implementation of Hadoop.

SHARE:

Just a month after insisting there was still a place for its own Hadoop competitor, Microsoft officials have decided to discontinue work on LINQ to HPC, codenamed "Dryad."

In a November 11 post on the Windows HPC Team Blog, officials said that Microsoft had provided a minor update to the latest test build of the Dryad code as part of Windows High Performance Computing (HPC) Pack 2008 R2 Service Pack (SP) 3. But they also noted that "this will be the final (Dryad) preview and we do not plan to move forward with a production release."

Dryad was supposed to provide a way for running big-data jobs across clusters of Windows servers. It was designed to provide a platform for developers to build applications that can process large amounts of unstructured data. Just a month ago, Microsoft updated its near-final test build of Dryad.

But it now appears Microsoft is putting all its big-data eggs in the Hadoop framework basket. Microsoft officials said a month ago that Microsoft was working with Hortonworks  to develop both a Windows Azure and a Windows Server distribution of Hadoop. A Community Technology Preview  (CTP) of the Windows Azure version is due out before the end of this calendar year; the Windows Server test build of Hadoop is due some time in 2012.

From the November 11 HPC Team blog post:

"Hadoop has emerged as a great platform for analyzing unstructured data or large volumes of data at low cost, which aligns well with Microsoft’s vision for its Information Platform.  It also has a vibrant community of users and developers eager to innovate on this platform. Microsoft is keen to not only contribute to this vibrant community, but also help its adoption in the Enterprise."

Microsoft Chairman Bill Gates first publicly mentioned Dryad, a Microsoft Research project, in 2006. The company took a number of steps to move Dryad from a research effort to a commercial one.

Microsoft’s stated longer-term goal was to combine LINQ to HPC and its parallel-programming tool stack to create an abstraction layer that would allow users to access compute resources — whether they’re located on multicore PCs, servers and/or the cloud. Microsoft officials said that Dryad/LINQ to HPC would be key to helping the company “turn the cloud into a supercomputer.”

In October this year, Microsoft officials said Microsoft's plan was to continue to work on various alternatives to Java-based Hadoop and MapReduce and that the company was  “still committed” to these efforts. I guess that's no longer the case.

I've asked company officials if there's anything more to say about Dryad. If there is, I will update this post.

Update: Microsoft officials won't say anything beyond what's in the blog post. So no word for those of you asking what Microsoft's plans are re: big data support for those with .Net experience. That was supposedly what differentiated Dryad from Hadoop -- at least according to the Softies a few weeks ago.

Topics: Operating Systems, Hardware, Microsoft, Servers, Software, Windows

About

Mary Jo has covered the tech industry for 30 years for a variety of publications and Web sites, and is a frequent guest on radio, TV and podcasts, speaking about all things Microsoft-related. She is the author of Microsoft 2.0: How Microsoft plans to stay relevant in the post-Gates era (John Wiley & Sons, 2008).

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

20 comments
Log in or register to join the discussion
  • RE: Microsoft drops Dryad; puts its big-data bets on Hadoop

    Seems sensible to me. What Microsoft has excelled at in the past is not creating new platforms, but taking existing ones and making them business-friendly.
    susannaking
    • RE: Microsoft drops Dryad; puts its big-data bets on Hadoop

      @susannaking@...

      are you serious? have you heard about win32, .net, asp? MSFT probably has created the world's most widely used platforms in history.
      neonspark
      • RE: Microsoft drops Dryad; puts its big-data bets on Hadoop

        @neonspark
        are you serious? have you herd about libc, java, html? Non Microsoft companies have create the world's most widely used platforms in history
        jessepollard
      • RE: Microsoft drops Dryad; puts its big-data bets on Hadoop

        @jessepollard -

        HTML isn't a platform - it's a simple page layout schema.

        Java isn't a platform - it's a programming language & runtime engine.

        Libc isn't a platform - it's the C runtime library.

        If you want to compare Microsoft's platforms, compare Windows NT and all successors (i.e. 2000, XP, Vista, 7, 8 along with their server variants) to GNU-Linux (HEAVILY derived from UNIX), BSD, OpenBSD (HEAVILY derived from UNIX), FreeBSD (HEAVILY derived from UNIX), OSX (HEAVILY derived from UNIX & incorporating NeXTStep).

        Out of those, which was the only brand new OS architecture created since 1989? NT.

        While it's true that Microsoft has had remarkable success in turning once bleeding-edge technology into tools and products that the general unwashed masses can not only use, but flourish with, is this such a bad thing?

        SQL Server was infinitely easier to use than Oracle, DB/2, etc. were at the time and delivered great results for most users. Exchange was infinitely easier to scale, deploy, manage and use than cc:Mail, Vines, Notes and Lantastic. Same for ActiveDirectory. Same for Office. Same for Windows.

        THAT is why Microsoft succeeded in putting a computer on every desk and in every home.
        bitcrazed
  • RE: Microsoft drops Dryad; puts its big-data bets on Hadoop

    this is dissapointing actually. As we have seen with HTML5 which evolves at a glacial pace and nobody can agree on anything, these platforms that are designed by focus groups always end up being a mess of convoluted ideas. I'd much rather have linq to hpc as an alternative to hadoop. maybe google will give us something that has a more clearly defined leadership and roadmap. oh well.
    neonspark
  • RE: Microsoft drops Dryad; puts its big-data bets on Hadoop

    The fact that it took almost a week before anyone commented on the drop has to be considered a pretty good indicator that there wasn't much interest in the product.
    curph
  • RE: Microsoft drops Dryad; puts its big-data bets on Hadoop

    I'd rather go with stable open source platforms that will always be there, not discontinued on a whim. I'm not against proprietary applications, but for platforms, you never know when they are just going to suddenly pull the rug out.
    deathjazz
  • RE: Microsoft drops Dryad; puts its big-data bets on Hadoop

    Dang, I was expecting this to be quite useful for certain informal but computationally intensive apps if it was ever finished. Does Hadoop have the same developer story?
    scH4MMER
  • Does this have anything to do with Yahoo!

    Yahoo! is Hadoop's major client -- they are essentially powered by Hadoop. Is there any chance that MS is getting ready to port Yahoo to Azure, as it were. Say, if they were to *own* them or something?
    x I'm tc
  • Support for Hadoop is the next to drop

    None of this big data frameworks is significantly useful. Cloud is not a sure thing to take off yet.
    LBiege
  • Doh!

    Another amazing (and superior) technology weighted done and killed by too much bureaucratic nonsense. It would have been great to have had an alternative to Hadoop. Will anyone save us from this elephant!?!? Maybe Splunk?
    msft_spelunker
    • RE: Microsoft drops Dryad; puts its big-data bets on Hadoop

      @msft_spelunker

      There are many Hadoop alternatives out there. The HPCC Systems platform is among them for tackling Big Data problems. Unlike Hadoop distributions which have only been available since 2009, HPCC is a mature platform, and provides for a data delivery engine together with a data transformation and linking system equivalent to Hadoop. The main advantages over other alternatives are the real-time delivery of data queries and the extremely powerful ECL language programming. For more information visit: http://hpccsystems.com
      H-M
    • RE: Microsoft drops Dryad; puts its big-data bets on Hadoop

      @msft_spelunker
      Amazing? Superior?
      Wow, a lot of vaporware.
      kirovs
  • RE: Microsoft drops Dryad; puts its big-data bets on Hadoop

    And this changes my life how? Oh, never mind...
    NoAxToGrind
  • RE: Microsoft drops Dryad; puts its big-data bets on Hadoop

    Reality is that Hadoop has such a great lead and mind share Microsoft has no choice but to embrace and try to catch up. Catching up for Microsoft will be tough because Hadoop is born and bred on Linux and that is where 100% of the early adopters are and that is the road that Enterprise is most likely to follow.
    katsnelson
  • RE: Microsoft drops Dryad; puts its big-data bets on Hadoop

    The qizmit project from MySpace was an attempt at hadoop in C# on windows - not much luck there. It raises the question of what they are planning to do - Java port or .net? Seems like they're damned if they do and damned if they don't. Also Hadoop is not just map-reduce, its a distributed data store, a distributed OS, People like Twitter are building other things on top of it like STORM (real-time event processing). Are Microsoft really going to give up that entire space to the open source community? Feels to me like someone hasn't understood all the issues - an apotheker moment perhaps?
    funazonki
    • RE: Microsoft drops Dryad; puts its big-data bets on Hadoop

      @funazonki

      They don't have much choice, just as they don't in HPC. Open source has done all this first, and there exists a requirement that MS be compatible. If they aren't, then nobody will use their stuff.
      jessepollard
  • tvomtfn 05 duq

    pfbcdx,wxeiiqxx21, tbsil.
    bmakrekdw80-24378989085206037169939486017350
  • RE: Microsoft drops Dryad; puts its big-data bets on Hadoop

    Simple question - couldn't you write C# code, compile it in mono, and run it using Hadoop Streaming?
    scott_blodgett
  • RE: Microsoft drops Dryad; puts its big-data bets on Hadoop

    @bitcrazed

    platform != Operating System

    all of win32, .net, asp, libc, java, & html, live *above* the operating system. the former trio could be run atop linux (via WINE and/or Mono), the latter atop windows (via Cygwin). So your comment completely misses the point.

    the list of libc, java, & html is intended as shorthand for the so-called LAMP stack, and thus to indicate the peer technology that was invented, before and outside of MS, that is equivalent to win32, .net, & asp. (tho, to be pedantic , JSP or CGI-bin -- to use the oldest examples -- are better equivalents to ASP. all three of these *produce* HTML.)

    i think what you meant to say is that MS was good for enterprise and/or desktop. fair enough.

    but, sorry, just couldn't let that other confusing misdirection stand.


    you're welcome.
    toddhodes