Microsoft makes available its Azure-based Hadoop service

Microsoft makes available its Azure-based Hadoop service

Summary: Microsoft's HDInsight service -- a cloud-based distribution of Hadoop -- is generally available. Microsoft's planned complement for Windows Server has been replaced.


Microsoft's cloud-based distribution of Hadoop -- which it has been developing for the past year-plus with Hortonworks -- is generally available as of October 28.


Microsoft officials also are acknowledging publicly that Microsoft has dropped plans to deliver a Microsoft-Hortonworks developed implementation of Windows Server, which was known as HDInsight Server for Windows. Instead, Microsoft will be advising customers who want Hadoop on Windows Server to go with Hortonworks Data Platform (HDP) for Windows.

Windows Azure HDInsight is "100 percent Apache Hadoop" and builds on top of HDP. HDInsight includes full compatibility with Apache Hadoop, as well as integration with Microsoft's own business-intelligence tools, such as Excel, SQL Server and PowerBI.

"Our vision is how do we bring big data to a billion people," said Eron Kelly, Microsoft's SQL Server General Manager. "We want to make the data and insights accessible to everyone." 

Making the Hadoop big-data framework available in the cloud, so that users can spin up and spin down Hadoop clusters when needed is one way Microsoft intends to meet this goal, Kelly said.

Microsoft and Hortonworks originally announced plans to bring the Hadoop big-data framework to Windows Server and Windows Azure in the fall of 2011. Microsoft made a first public preview of its Hadoop on Windows Server product (known officially as HDInsight Server for Windows) available in October 2012.

Microsoft made available its first public preview of its Hadoop on Windows Azure service, known as HDInsight Service, on March 18. Before that, it had made a number of private previews available to select testers.

Hortonworks announced in February this year that its HDP platform was an extension of its two-year-old Hadoop partnership with Microsoft. (HDP didn't exist yet when Microsoft and Hortonworks initially announced their partnership.)

HDP allows users to deploy Hadoop on Windows Server in their own datacenters -- the same way they can already deploy HDP on several Linux distributions. Microsoft and Hortonworks are touting HDP for Windows as offering an easy migration path to HDInsight. 

Hortonworks announced general availability of HDP 2.0 last week. Hortonworks will update HDP for Windows Server to take advantage of HDP 2.0 within a month, Kelly said. The pair haven't committed to a timetable as to when they'll update HDInsight to take advantage of HDP 2.0. (It's currently on version 1.3.)

As part of its Hortonworks partnership, Microsoft has contributed back to Apache 25,000 lines of code for inclusion in the Apache Hadoop code base, Kelly said. 

Topics: Cloud, Big Data, Microsoft, Windows Server


Mary Jo has covered the tech industry for 30 years for a variety of publications and Web sites, and is a frequent guest on radio, TV and podcasts, speaking about all things Microsoft-related. She is the author of Microsoft 2.0: How Microsoft plans to stay relevant in the post-Gates era (John Wiley & Sons, 2008).

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • Makes sense, as far as ease of use goes

    But the big problem, as always, is how do you downlevel the data sets in the first place? Moving the recordsets such that you can run analysis on them is the real problem. May only be practical if you grow the data in the cloud in the first place (which eventually will happen.)
    • Hadoop is not as useful as advertised

      Notice how Yahoo failed to make much money from Hadoop despite heavy usage of it for what 7-8 years? Being Yahoo they sure have plenty data (from their web pages) to analyse. If they cannot monetize off it after so many years why would anyone else succeed at it?
      • Yahoo do make money from Big Data

        or more precisely, Yahoo do use Big Data to make money. Their business model is not about selling technologies, like Cloudera/MapR. They are a media company. Big difference. Say what you will about Yahoo, just keep in mind that they are still #4 web property. They could not run their operations w/o Hadoop. how do we know? We aggregate daily Big Data news from all major industry sources at
    • Makes little sense to me

      Mac - exactly that's the point: Big Data upload in the cloud? Leave the technical problems with uploading petabytes. Leave the financial complications as well. Just answer WHAT KIND of datasets these are?
      Customers, patients, banking. Mostly.
      Are all the bolts still in place?
  • Why would I care about Azure?

    Like everything from Microsoft it's just an abomination of other open, public protocols and code.

    Microsoft = steal everything for free and then charge for it!
    • I've found it to be

      quite easy to use to run virtual machines to execute test of hpc code. Ultimately the code get run on more powerful computers, but it works well for testing and keeps my workstation free. Some of the amazon services work well too, but Microsoft has made it very easy/simple to set up virtual machines in azure.
      Sam Wagner
    • All software is protocols and code.

      Without protocols, your code ends up jumbled.

      Without code, your program/application doesn't do anything.

      Oh, and if Microsoft was "steal everything for free and then charge for it", why is Visual Studio Express free?

      Hell, if you're a developer, you should be thanking them.

      Without Microsoft, you probably wouldn't even own a PC.
    • Uh, that's the typical open-source company's strategem

      Look at IBM, Red Hat, etc. The get the Linux kernel and other FOSS software, package it up, provide service for it, and sell the package and services.

      Microsoft creates an awful lot of it's own stuff (Windows, most of it's server products, Visual Studio). I'm not sure where you're coming from.
  • Might be useful

    Maybe a mention of where MapReduce comes from?,650,331.PN.&OS=PN/7,650,331&RS=PN/7,650,331
  • HDInsight

    HDInsight is the first step by MSFT, to establish itself in the Big Data Space. In all totality of the complete stack MSFT misses the bus. I have gone from HDInsight to cloudera Impala on Azure please find my experience here
    Ajay Solanki