Microsoft's road to the cloud is paved with parallelism

Microsoft's road to the cloud is paved with parallelism

Summary: A new whitepaper that Microsoft researchers are set to present at a conference next month sheds more light on Microsoft's back-end cloud infrastructure.

SHARE:

A new whitepaper that Microsoft researchers are set to present at a conference next month sheds more light on Microsoft's back-end cloud infrastructure.

Microsoft’s road to the cloud is paved with parallelismThe paper, entitled, "SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets," details a new declarative scripting language that is optimized for storing and analyzing massive data sets (like search logs and click streams) that are key to cloud-scale service architectures.  SCOPE, or Structure Computations Optimized for Parallel Execution, is the name of the language.

According to the paper -- which Microsoft is on tap to present at the VLDB 2008 conference in late August -- SCOPE doesn't require explicit parallelism, but it will be "amenable to efficient parallel execution" across large clusters. SCOPE is like SQL, but with C# extensions, the paper says.

I found the new whitepaper via a blog link from Greg Linden, an employee of Microsoft's Live Labs. Linden blogged:

"Scope is similar to Yahoo's Pig, which is a higher level language on top of Hadoop, or Google's Sawzall, which is a higher level language on top of MapReduce. But, where Pig focuses on and advocates a more imperative programming style, Scope looks much more like SQL."

Reading through the paper, I noticed an explanation of how SCOPE fits in with Cosmos, Microsoft's back-end storage layer that currently powers Live Search and other Microsoft services. The SCOPE whitepaper sheds more light on what Cosmos is and how it works. From the paper:

"Microsoft has developed a distributed computing platform, called Cosmos, for storing and analyzing massive data sets. Cosmos is designed to run on large clusters consisting of thousands of commodity servers. Disk storage is distributed with each server having one or more direct-attached disks."

(A loosely-coupled aside: I wonder if Pat Helland's decision to move to the SQL team at Microsoft has any connection to all of this. Helland's expertise is in big-picture strategy around transactional and parallel processing, as well as service-oriented architectures.)

Increasingly, all of Microsoft's future strategies and products finally seem to be converging. More teams are thinking about parallel/distributed/multicore computing, with the experimental Windows successor code-named Midori being just the most recent of many examples. More Microsoft products are seemingly being designed with modeling in mind from the get-go.

Maybe Chief Software Architect Ray Ozzie's campaign to break "drive alignment" across the various Microsoft product groups is finally taking root.... Or maybe it's simply that cloud computing, to be truly scalable, must be built to work across increasingly large networks of distributed systems. Or maybe it's a little of both....

Topics: CXO, Cloud, Data Centers, Data Management, Enterprise Software, Microsoft, Software, Software Development

About

Mary Jo has covered the tech industry for 30 years for a variety of publications and Web sites, and is a frequent guest on radio, TV and podcasts, speaking about all things Microsoft-related. She is the author of Microsoft 2.0: How Microsoft plans to stay relevant in the post-Gates era (John Wiley & Sons, 2008).

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

7 comments
Log in or register to join the discussion
  • More Investment

    Two words. How long?

    This new commitment is certainly a good direction, but it's
    hard to see it as leadership at this late stage. The move
    into parallelism is a great equalizer. The laws of
    diminishing returns are applied across the boards. Legacy
    software will require rewrites. The requirement to reinvest
    in parallel aware software, and or abandon software for
    services, will break Microsoft's lock on many businesses.
    Marketplace balance has not traditionally been Microsoft's
    friend.

    The new standard ? post Vista - is shipping software, not
    this weeks codeword. If the Vista stumble leads to
    overcompensation and another round of unduly raised
    expectations, it will be a drastic mistake. Confidence is
    already waning.

    If parallelism is stagnant or late relative to competitors,
    Microsoft can no longer count on it's past ability to avoid
    competition by hosting competition. It can no longer count
    on an aura of inevitability.

    The commitment, or lack of it, is being played out in the
    game console space where Microsoft's short term strategy
    of multiple processors operating within traditional
    programming models led the XBox 360 to a quick lead.
    The PS3 however, with it's multiple cores and it's difficult
    but forward-thinking programming model, has cordoned
    off more room for growth. The gap has begun to narrow.
    As it turns out, this wasn't a sprint.

    Meanwhile, OSX Snow Leopard has made a commitment to
    parallelism, recovering efficiencies, and scalability both up
    and down. The iPhone is a physical manifestation of what
    seems to be advancements to the consumer. Regardless
    what arguments anyone has?it's an effective magic trick
    that ? to invert Arthur C. Clarke ? is indistinguishable from
    technology. Codewords will not compete with this device.

    More investment is now required for less return. The gold
    rush is over. A marketplace that demands growth may look
    elsewhere for it.
    Harry Bardal
    • I'm pretty sure...

      I heard the new X-Files film is out as well. I might go see it tonight.
      GTRoberts100
    • Interesting......one sided as usual...but interesting..

      Harry, you've said these same things for years. why not wait until just one of your predictions comes true before making another set of them? Wouldn't that be wise? <br>
      Seems no matter how many times you've said sony is overtaking MS with the PS3, it still has not happened. <br><br>
      And the snow Leopard thing...sorry but just because after 25 years Apple was wise enough to move to the dreaded x86 platform while MS was mired in sanctions from civil suits ....why don't we save the "snow Leopard" comments for the day, year perhaps it's more than your marketing bytes and there is actually something to talk about. <br><br>
      I think Apple is going to find it's quick strike tactics for short term gains is nothing more than the team coming late to the game, with hopes of winning yesterday's prize. <br><br>
      xuniL_z
      • It's not that parallelism is a new idea

        Microsoft is fleshing out their version of what Yahoo already does. Though this might solve some of the mystery about why Microsoft so badly wanted them beyond the search chops.

        As you say Apple is trying for a fast strike into the space with, no doubt, a lot of cheering from the faithful when Jobs announces it.

        Linux has, in one form or another, been at it for a few years now, quietly due to kernel improvements coming from the likes of Google and IBM. For that matter, so has Solaris.

        Let's face it, what we have here is a research paper being presented on how Microsoft [i]might[/i] do it and some good ideas contained in it whether they're copying the ideas from someone else or not.

        It does have the scent of a PR move to it, and worse, as Mary Jo presents it, a vapourware announcement.

        Though, to be perfectly honest, parallelism is just a new name for parallel processing which has been around since the ancient mainframe days. All we have here is a model of what Microsoft might do with it not what they will do. And, at least, one of the MS faithful has already posted a screaming TalkBack about it's wonders.

        All that said I wish them luck. Competing models in the space be they from MS, Google or whomever will mean that we on the consuming end will be better off in the long run.

        Maybe Microsoft can actually get the Internet right this time.

        ttfn

        John
        TtfnJohn
        • They never claimed it to be anything more than conceptual ....

          at this point...why are bloggers taking a white paper and making news out of it is the real question. <br><br>
          I find people are all too quick to act as a MS spokesperson and turn something not blog worthy into another lecture on how Microsoft is yet again screwing up, not having the goods...whatever. Why do we need that from something that wasn't ever presented in a way that comments are neccessy. <br>I've read a certain apple fan's posts enough to see him or her start a diatribe on the failings or "control" of MS, come up with something unfair MS could do and in a stream of consciousness ( stream of unconsciousness maybe) style start to chastise them for the act they had not committed and only had been conjoured up by the poster! He got so involved in what they "could" do, it became as though they had, and we got treated to how low down that was......MS makes people unnecessarily crazy.
          xuniL_z
  • Prediction

    LIVE MESH IS GOING TO BE INTEGRATED WITH MIDORI. SOON MAJORITY OF SOFTWARE IS GOING TO BE HOSTED ONLINE. I BET MICROSOFT IS BUILDING DATA CENTERS FOR SMALL DEVELOPERS WHO CANNOT AFFORD TO TO HOST THE BANDWIDTH REQUIRED TO RUN THEIR SOFTWARE. THE HARD DRIVE IS GOING TO COMPLEMENT SAAS. CONSUMERS AND BUSINESS STILL IS GOING TO WANT PEACE OF MIND OF HAVING AN BACKUP OF THEIR SOFTWARE.
    1g2j
  • Ah, yes subscriptions and services...

    I remember back in the stone age when cable was born. Then it was, well, yes you are paying for it but NO COMMERCIALS, WOW!

    Years have passed and in most markets not only are you paying monthly, but there are programs everywhere on every cable station AND you are paying EVEN FOR INFO-MERCIALS. You are paying for THEM to ADVERTISE to YOU, DAH, how stupid.

    I often hear conversations about how cable sucks and how people feel helpless. I remind them of the fact that they can just turn it off. If enough people turn it off things will change.

    So as far as I am concerned all companies that work on subscription like cable, XM radio, or companies that provide software as a service, operating in the cloud etc. etc. can put it where the sun doesn't shine.

    I dislike this "model" so much that I am learning how to use LINUX because Microsoft and even Apple are moving to this model and I can cut them loose.

    I have a unique perspective since I have worked at companies like Microsoft and I will be damned if I will be some cash cow for all those lazy, fat ass programmers etc. and allow these big companies to bleed off my cash.

    I am not addicted to email, TV, radio etc. so if this is the wave of the future I will just "cut the wire" and read more, take more walks, etc. etc. I am old enough to remember a day when there wasn't any such thing as TV or computers and radio was a hour or two of entertainment in the evening.

    Feel free to pass this on, I suspect there are many people like me and the egg heads who live in their plugged in incestuous little world haven't figured this out yet.
    mikifinaz1@...