Windows Azure futures: Turning the cloud into a supercomputer

Windows Azure futures: Turning the cloud into a supercomputer

Summary: I haven't written much about longer-term Azure futures. Until this week, which just so happens to be the one-year anniversary of Microsoft's cloud platform.

SHARE:

February 1 is considered the "one year" anniversary of Microsoft's Azure cloud platform (even though February 2 is the actual date that billing was "turned on").

Last year, Microsoft said it had 10,000 Azure customers; this week officials are saying they have 31,000, though they are refusing to say how many of these are paying customers, how many are divisions of Microsoft, etc.

As I noted last year, Microsoft has been slowly and steadily adding new features to Azure. But I haven't written much about longer-term Azure futures. Until today.

Bill Hilf, General Manager of the Technical Computing Group (TCG) at Microsoft, isn't part of the Azure team. But he and his band are doing work on technologies that ultimately may have substantial bearing on the future of Microsoft's cloud platform. The TCG has a server operating system team, a parallelization team and a team "with the idea of connecting a consumer to a cloud service," according to Hilf.

The TCG late last year stated its intentions to allow customers to provision and manage Windows Server 2008 R2 HPC nodes in Windows Azure from within on-premises server clusters as part of Service Pack 1 of HPC Server 2008 R2. But Hilf and his team want to go beyond this and turn the cloud into a supercomputer, as Hilf explained to me last week. "We want to take HPC out of niche access," he said.

This isn't going to happen overnight, even though the biggest Azure customers today are the ones using HPC on-premises at the current time, Hilf said. HPC and "media" (like the rendering done by customers like Pixar) are currently the biggest workloads for the cloud, Hilf said.

To bridge HPC and Azure, Hilf has a multi-pronged strategy in mind. One of the prongs is Dryad. Dryad is Microsoft’s competitor to Google MapReduce and Apache Hadoop. In the early phase of its existence, Dryad was a Microsoft Research project dedicated to developing ways to write parallel and distributed programs that can scale from small clusters to large datacenters. Both the Bing and the Xbox Live teams have used Dryad in building their back-end datacenters.

There’s a DryadLINQ compiler and runtime that is related to the project. Microsoft released builds of Dryad and DryadLINQ code to academics for noncommercial use in the summer 2009. Microsoft moved Dryad from its research to its Technical Computing Group this year.

"Dryad, in its first iteration, is really for on-premises," Hilf told me during an interview last week. "Eventually, we'll roll Dryad up into Azure, as even more data is put in the cloud."

Go to the next page for more on how Microsoft's parallel stack comes into play

Before that happens, Microsoft will include Dryad in Service Pack 2 of HPC Server 2008 R2, Hilf said. He didn't have a firm ship-target, but hinted "late 2011" might not be a farfetched timeframe. (Microsoft announced last week that it also is making the Dryad code available to interested testers as one of the technologies in its newly formed TC Labs subsection of DevLabs. (TC Labs will be for "any software that is not yet in a product, and of which all or none of it might become part of a product" at some point, Hilf explained.)

Another TCG prong which could have impact on Azure's future is what Hilf is calling the "parallelization stack." This stack includes the runtimes, languages and other parallel/multicore tools and technolgies that Microsoft has been building for the past couple of years. This parallel-computing layer/platform will be rebranded around the time of the Professional Developers Conference, Hilf said (which I hear is happening in the early fall of 2011).

It's the combination of these technologies -- including Dryad and the parallel stack -- which will enable Microsoft to create an abstraction layer that will allow users to access compute resources -- whether they're on multicore PCs, servers and/or the cloud, Hilf said. The customers most likely to benefit are the real data wonks -- the folks that Hilf calls "domain specialists." These are the individuals in financial services, manufacturing, oil and gas, media, the hard sciences, and other data-intensive professions who have an insatiable appetite for data.

It's still early days for this kind of supercomputer-in-the-sky stuff. But Hilf already is talking up things like a technical-computing PaaS (platform as a service) and TC SaaS (software as a service).

I've got to say that Hilf's whiteboarding got me wondering whether Azure will remain Microsoft's core cloud PaaS in the longer term, or if something like the still-mysterious Midori operating system could come into play here. As the M-word (Midori) is not allowed to be uttered in the hallowed Redmond halls by us bloggers/journalists, I didn't even attempt to ask. I did ask about Orleans -- a future cloud programming model technology that's being pioneered in the more research-centric Microsoft eXtreme Computing Group -- and was told by Hilf that it looked interesting in concept, but that it was too early to say if/how/when it could come into play.

My biggest take-away from my interview with Hilf is Microsoft isn't waiting around for its business customers to overcome their cloud objections. Microsoft is looking for other ways to attract enterprise customers to the cloud, with nearly unlimited data-access and processing power as the lure.

(image credit: @davidlowe)

Topics: Microsoft, CXO, IT Employment, Windows

About

Mary Jo has covered the tech industry for 30 years for a variety of publications and Web sites, and is a frequent guest on radio, TV and podcasts, speaking about all things Microsoft-related. She is the author of Microsoft 2.0: How Microsoft plans to stay relevant in the post-Gates era (John Wiley & Sons, 2008).

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

33 comments
Log in or register to join the discussion
  • RE: Windows Azure futures: Turning the cloud into a supercomputer

    Azure seems to be very quietly making significant inroads and I suspect it is going to be huge technology. WP7 and Azure are an intriguing combo also.
    Mythos7
  • I think that is the right idea

    I believe as a general rule of thumb: if a task on a PC takes 1 or more hours to complete, consider offering to do the task via a cloud service. This should be the attitude of ISVs that create video processing software, development packages such as FORTRAN, etc. These guys should be offering from within their applications, to have certain data processing take place in the cloud, in a fraction of the time it normally takes on a PC, or alone on company servers.

    Also instead of threatening the jobs of IT guys with the idea of gutting IT departments, and replacing them with public clouds, help these guys out by offering a slew of public cloud services, that make their lives easier, and allow them and the companies they work for, to do things they couldn't do before, due to lack of funds, resources, etc.
    P. Douglas
    • RE: Windows Azure futures: Turning the cloud into a supercomputer

      @P. Douglas: Great call on the IT cloud services. IMHO, the cloud offers the greatest benefit to the small to mid size businesses or IT consultants. Small busnisses either hire a 3rd party consultant firm (usually a small local one) or have an extremely tech savvy in-house guy to handle their IT for them. Either way, the small business doesn't have the financial ability to use enterprise level cloud services or IT infrastructure, so your point is spot on.
      Qube-It
      • RE: Windows Azure futures: Turning the cloud into a supercomputer

        Information is truly extraordinary, I am very interested to read this post..<a href="http://www.universaldegrees.com/universaldegrees/programs/bachelors-degree-program.asp">online bachelors degree</a>
        <a href="http://www.universaldegrees.com/universaldegrees/programs/doctorate-degree-program.asp">online doctorate degree</a>
        <a href="http://www.universaldegrees.com/universaldegrees/programs/associate-degree-program.asp">online associates degree</a>
        <a href="http://www.universaldegrees.com/universaldegrees/programs/undergraduate-certificate-program.asp">online undergraduate certificate</a>
        Shamrock12
      • RE: Windows Azure futures: Turning the cloud into a supercomputer

        Be thankful for that.
        <a href="http://iibps.co.in/accredition.aspx">IAO</a>
        <a href="http://www.gesfwa.org/gesfwa/news2.asp">IAO Accreditation</a>
        JonathanTrott
      • RE: Windows Azure futures: Turning the cloud into a supercomputer

        I adore what you have got right here.
        <a href="http://priorlearninghorizons.com/">WEVAC University</a>
        JonathanTrott
      • RE: Windows Azure futures: Turning the cloud into a supercomputer

        @Qube-It Nice Things in your content which i need.....Thanks for that...
        [url=http://www.mightydesigners.com]Logo Design Contest[/url]
        astonatkin
    • RE: Windows Azure futures: Turning the cloud into a supercomputer

      These guys should be offering from within their applications, to have certain data processing take place in the cloud, in a fraction of the time it normally takes on a PC, or alone on company servers. <a href="http://educationfuturist.wordpress.com/">Belford university</a>
      <a href="http://onlinediplomahelp.com/?p=38">accredited high school diploma</a>
      JonathanTrott
      • RE: Windows Azure futures: Turning the cloud into a supercomputer

        @JonathanTrott I read many blogs related to windows but i don't like any one...but your discussion about window is relevant and different to other that's why i like your blog...keep it up!
        [url=http://www.writers-write.co.uk/custom-assignment-help.php]Buy Assignments Help Online[/url]||[url=http://www.writers-write.co.uk/custom-coursework.php]Coursework Help UK Online[/url]||[url=http://www.writers-write.co.uk/dissertation-writing-help.php]Dissertation Writing Service UK[/url]||[url=http://www.writers-write.co.uk/Article-Assignment.php]UK Assignment Writing Services Online[/url]||[url=http://www.writers-write.co.uk/article-buy-essay.php]Buy UK Essays Online[/url]||[url=http://www.writers-write.co.uk/statistics-help.php]Statistics Help[/url]
        jaziblee
  • Pixar is a reference customer...

    Got to love the irony.

    Pixar, previously owned by Steve Jobs, now owned by Disney with Steve as the #1 shareholder, using Microsoft's cloud for movie rendering...
    dazzlingd
    • RE: Windows Azure futures: Turning the cloud into a supercomputer

      @dazzlingd <br>Last time I checked Pixar had a huge Linux farm and some Macs running Renderman. There are actually job posts for Linux render farms admins from last year.<br>I know they did POC with Azure, but that does not mean they use it for production. Does anyone have more precise information? Perhaps they do not like Amazon, because this would be the natural place for Renderman to be.<br>That brings another point- it will be challenging for Azure to compete, because it has no Linux option unlike Amazon and others. I have heard some future plans for virtualization and running Linux images, perhaps someone knows more about that.<br>Finally, Azure is well behind Amazon in terms of public sets (for example 1000 genomes, Ensembl, etc.). They should do better if they expect to stay relevant in the science area.
      kirovs@...
      • Pixar use Azure for on-demand compute cycles

        @kirovs@... I don't know how this ties into their Linux farm, but according to the press release on the Microsoft website, Pixar use Azure for on-demand compute cycles using their RenderMan platform.

        I also remember seeing a video showing how James Cameron used Azure to render the CGI scenes in Avatar. I believe he uses RenderMan too, so there must be a link somewhere between both platforms.

        Interestingly enough, if you search for Pixar and Microsoft you will find several links to projects that Pixar and Microsoft Research have done together which shows a long established relationship.

        I guess that puts to rest the old meme of Microsoft and Steve Jobs hate each other. That will disappoint a lot of the deluded fanboys from both camps...
        dazzlingd
      • RE: Windows Azure futures: Turning the cloud into a supercomputer

        @dazzlingd@... <br>What you say make things even more complicated in my opinion- Avatar was rendered on a supercomputer running RedHat (or Ubunutu?) Linux in New Zealand (WETA Digital). Here is a link:<br><a href="http://www.junauza.com/2010/01/technology-behind-avatar-movie.html" target="_blank" rel="nofollow">http://www.junauza.com/2010/01/technology-behind-avatar-movie.html</a><br>So I am still not quite sure where Azure comes into the picture...
        But thanks for the explanation- I guess when there is peak demand they move load to Azure...
        kirovs@...
  • Azure is a winner for-zure....

    A few years ago, I put some of my top, Top, TOP MCSD's on this Azure project. I had them working 24x7x365 getting our on premises applications cloud enabled. Once we deployed the apps to the cloud, I fired all of them and saw immediate ROI. "To the cloud!".
    Mike Cox
    • RE: Windows Azure futures: Turning the cloud into a supercomputer

      @Mike Cox

      Is this the real you? Where is your rep?
      The one and only, Cylon Centurion
  • TRUE!

    The cloud is the future people!




    source:
    fakesteveballmer.blogspot.com
    nomoreds
  • RE: Windows Azure futures: Turning the cloud into a supercomputer

    dumb dumb the cloud is not in the usa it is over seas not one is in the usa and it will not save your files on your computer at all and it has been hacked already it has been on tv news for mouths and microsoft is not doing any thing about it AGAIN YOUR FILES ARE NOT SAVED ON YOUR COMPUTER THEY ARE SAVED OVER SEAS SOME WHERE AND YOU CAN NOT TELL WHERE AT ALL AND ANY ONE CAN ACCESS YOUR FILES AT WILL THE NEWS SAID HACKERS GET IN EACH DAY SO YOU CAN BELIV THE MICROSOFT AND OTHERS THAT IT IS SAFE ALL YOU WOULD AND IF YOU LOSE YOUR PASSWORD KISS YOUR FILES AND PIC'S GOODBY FOR GOOD
    ttx19
  • RE: Windows Azure futures: Turning the cloud into a supercomputer

    Very interesting and informative blog. Hope we get some updates<a href="http://www.minmetalschina.com/about.html">Die Steel</a>
    flypig0089
    • RE: Windows Azure futures: Turning the cloud into a supercomputer

      @flypig0089 i agree with you... Mary good article i really love it. I have reading this article is awesome so good work.... Keep working.... [url=http://www.allwritingsource.co.uk/]Writing Services[/url] || [url=http://www.allwritingsource.co.uk/essay-help.php]UK Essay Help[/url] || [url=http://www.allwritingsource.co.uk/essay-writing-service.php]Essay Writing Service[/url]
      adokadrik
  • RE: Windows Azure futures: Turning the cloud into a supercomputer

    Which projected does one assume if you find yourself providing software solely spot ? I simply definitively much like the carry out. Many thanks for this current. [url=http://www.nflplan.com/]pittsburgh steelers jerseys[/url],[url=http://www.nflplan.com/michael-vick-jersey-medium-michael-vick-jersey]michael vick jersey[/url],[url=http://www.nflplan.com/kids-peyton-manning-jersey-peyton-manning-jersey]peyton manning jersey[/url],[url=http://www.nflplan.com/pink-walter-payton-jersey-kids-walter-payton-jersey]walter payton jersey[/url],[url=http://www.nflplan.com/cheap-desean-jackson-jersey-desean-jackson-jersey-philadelphia]desean jackson jersey[/url], I've got would allow you to gram calorie take note of, you'd land up i am only everyplace doing this. I managed to get two or three your individual page having all sorts of world-wide-web sheet and was indeed the truth is obsessed with begin amount of desire and even thinking about increased.
    dfwekrdfe35-24353603514803697860648270628153