Red Dog: Can you teach old Windows hounds new tricks?

Red Dog: Can you teach old Windows hounds new tricks?

Summary: What led Microsoft -- a company that has spent a good part of the past decade-plus protecting the Windows franchise at the expense of the Web -- to finally create an infrastructure that would support not just Windows developers, but also Web programmers?

SHARE:

It has been four months since Microsoft took the official wraps off its cloud-computing initiative. Yet still relatively little still is known about the Azure platform and plans.

The part of Azure which intrigued me the most was the cloud operating system, code-named "Red Dog," that is at its heart. Late last month, Microsoft allowed me access to many of the principals behind Red Dog -- everyone from the infamous father of VMS and NT, David Cutler, to the handful of top-dog engineers who helped design and develop the various Red Dog core components. Over the course of this week, I'm going to be publishing a post a day about Red Dog.

Part 1: It's not just about Windows any more

What led Microsoft -- which has spent a good part of the past decade-plus protecting the Windows franchise at the expense of the Web -- to finally create an infrastructure that would support not just Windows developers, but also Web programmers? And how did a company known for its slipping dates more than making its shipping dates manage to build a cloud-computing platform that developers could begin test-driving in less than two years?

Not so long ago, Microsoft probably would have simply rented out a bunch of Windows Server machines and expected that anyone inside or outside the company interested in making use of them would flock to pay for datacenter power by the hour. The Microsoft of old would have pitted multiple in-house development teams (unbeknown to each of them) against one another in designing the various cloud-computing components, with "the best" team ultimately winning. And the good ol' Soft would have, undoubtedly, counted on having four or five years to get its act together before even thinking about fielding a first cloud OS test build.

None of that happened with Red Dog. Even before the finishing touches were done on Windows Vista, CEO Steve Ballmer was talking with Amitabh Srivastava, Corporate Vice President, about his next assignment. At that point, Srivastava, a 12-year Microsoft veteran, was a leader in the Windows team and was in charge of redesigning the engineering processes around how Windows was built.

In 2006, "I was thinking about what to do next," Srivastava said. "Would I work on the next version of Windows?"

Chief Software Architect Ray Ozzie had joined Microsoft about a year before and "was completely high on services," Srivastava recalled. "But I didn't have a services background. Steve (Ballmer) suggested I go talk to Ray." Hours later, the pair were still talking.

[How to turn a bunch of Windows guys into services guys] -->

Srivastava said he came away with a couple of key realizations:

1. Nobody really knows services. "Everyone" -- even "thought leaders" like Amazon and Google -- "are just five minutes into the first quarter," Srivastava quipped.

2. In spite of this fact, Microsoft actually had some substantial services experience, but not in Windows. Microsoft had been running services like Hotmail for more than ten years, Srivastava said, and there were more than 160 identifiable services runing in various units throughout the company. So even if Srivastava were to do no more than create a unified infrastructure for Microsoft's own internal services, he'd still help the company realize huge cost savings, he said.

Srivasta coaxed Dave Cutler, the father of Microsoft's NT operating system, out of "semi-retirement," and began assembling a core team of engineers. Because of Srivastava's and Cutler's pedigrees, the team consisted primarily of "Windows guys."

So how do you turn a bunch of Windows guys into services guys?

The first order of business for the core team was educating themselves about the services world. The team rented a van and drove around Redmond and Silicon Valley for two-plus months to talk to the various teams at Microsoft that already had services know-how. They talked to the Hotmail team, the Virtual Earth team, the Xbox Live team. They asked each unit about their Windows' pain points and solicited wish lists.

"We went to one of Microsoft's datacenters," Srivastava reminisced. "We all became ops guys for a day."

(It was during this tour that the "Red Dog" project got its code name. The inspiration? A seedy club in Silicon Valley called the "Pink Poodle.")

The team quickly realized that while Red Dog would be a technical project, most of the problems it would need to solve were business ones, according to Todd Proebsting, Director, Technical Strategy, Windows Azure, and one of the original Red Dog core team.

"We had a euphoria stage, where we felt like we could do anything," Proebsting said. "Then we narrowed (the ideas) down, after talking to customers," many of whom worked at Microsoft on various teams. "We talked to Search and we talked to CarPoint," he said. "We asked them what we could do for them" and took note of the pieces that those teams already had developed which could be reused with Red Dog.

The findings: Cutting costs was essential. Providing greater levels of reliability was essential.

"If we could get the number of machines down or reduce the amount of required support staff," that was going to be huge," Proebsting said. The potential Red Dog customers wanted to know how long it would take for a new feature to be introduced or to address an uptick in server demand. Those were their hot buttons, he said.

Creating a 'services mindset'

"Our problem was how do you create a services mindset?" Srivastava said. Instead of spending years crafting an operating system and then deploying it, the team needed to think about writing pieces of software that would be deployed immediately.

The team published all of its members' cell phone numbers, so the Red Doggers had an immediate understanding of what 24X7 and "five nines" (99.999 percent) availability really meant.

Srivastava, with input from the core Red Dog group, authored a memo called "Owning the Cloud," which layed out their plan of attack. (Microsoft declined to make the memo available, claiming it includes proprietary information.) At this point, the team realized the key to the cloud was to be able to better manage the datacenter.

"The idea became managing a datacenter as an operating system," Srivastava said. "We wanted to abstract the whole thing and manage all the resources."

[Ballmer: 'Go, Go, Go!'] -->

"We went in with a completely clean slate, and a lot of that credit goes to Ray (Ozzie)," Srivastava said. "We didn't want to just go and do another new Windows API (application programming interface. It wasn't about protecting Windows -- it was more about leveraging it."

In October 2006, Srivastava & Co. sent their vision statement to Ballmer. In November, Ballmer sent back mail that said "Go, Go, Go!" Srivastava said. A tight core team of about 20 senior engineers was assembled -- nothing like the size and scope of the thousands of folks working on Windows these days -- and off they went.

"Almost nobody inside Microsoft knew about Red Dog," Srivastava said. (Including Chairman Bill Gates, he noted.) "And none of us (on the team) knew if it was going to work."

The Red Dog team realized quickly that they couldn't ask any of the existing teams at Microsoft to take a chance on them by using their fledgling infrastructure, so they decided they needed to build their own datacenter test bed. The team commandeered some space in Building 18 and built a mini datacenter that was 1/20th the size of a full-fledged one. The team "took" the power it needed to run the center from three other buildings. Some of the other teams in 18 wondered what was going on in the offices next door. There were a few noise complaints, Srivastava recalled.

In April 2007, the team began writing code for the components it needed to run Windows Server 2008-powered datacenters: The fabric controller, a storage system, a virtualation substrate and a development environment. By November 2007, the first version of Red Dog was working.

(Watch for Tuesday's installment: Microsoft's Cloud OS Dream Team: A Who's Who)

Topics: CXO, Data Centers, Microsoft, Operating Systems, Software, Windows

About

Mary Jo has covered the tech industry for 30 years for a variety of publications and Web sites, and is a frequent guest on radio, TV and podcasts, speaking about all things Microsoft-related. She is the author of Microsoft 2.0: How Microsoft plans to stay relevant in the post-Gates era (John Wiley & Sons, 2008).

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

64 comments
Log in or register to join the discussion
  • Obviously, Microsoft has a lot of money to invest, and they have now

    decided it is time. Will be interesting to see what they actually create.
    DonnieBoy
    • Just alpha hypeware lock-in encumbered nonsense.

      It's Microsoft we're talking about, what else do they do except try to use software to dip their dirty little fingers into your bank account?
      Amelioration
  • New tricks for an old dog?

    No. Just recycling an old one. I see they've retrieved the old Office "helper" dog, painted him red and given him a new assignment.
    kozmcrae
    • They don't have a chance.

      Microsoft doesn't know how to compete, they couldn't even compete with XP.

      This is the tech version of GM meet Toyota...

      In the long run it will be a better world.
      zamzmith
      • What I don't Understand: First Thing

        The first thing I don't understand is why anyone, except in very limited circumstances, would want to use Cloud Computing?

        Why would anyone want to hang their most sensitive data out to dry on someone else's clothes line? Please don't tell me that the data would be encrypted and safe from prying eyes. These companies, MS, Google, et al, have made their fortunes by amassing and selling other people's data and the statistics about other peoples data. Don't tell me they'd stop now just when they would have it ALL!

        This is not to mention what AT&T, Google and others have shown us in their dealings with the Chinese: If you hand your eMail and data over to the big boys, you can forget due process, subpoena, presumption of innocence and all your other constitutional rights. They will hand all your files AND THE ENCRYPTION KEYS over to Big Brother faster than you can say Habeas Corpus.

        Remember, Clouds are where the Sun don't shine!

        copyright Western News Co (Chicago) by permission of author.

        <a href="http://talkback.zdnet.com/5208-12558-0.html?forumID=1&threadID=53543&messageID=1126876">What I don't Understand: Second Thing </a>

        <a href="http://talkback.zdnet.com/5208-12558-0.html?forumID=1&threadID=55543&messageID=1126904">What I don't Understand: Third Thing </a>

        <a href="http://www.westernnewsco.com">Western News Co (Chicago)</a><br>
        Seamus O'Brog
        • You are looking at the cloud as a free service

          If it's a for fee service (like Amazon) then they won't do things like that. I doubt that all the businesses hosted by Amazon are sharing their client lists...
          914four
    • You claerly haven't done your reading

      It's hard to say whether Red Dog/Windows Azure will work when it is widely available but the concepts they're using and the current implementation (live today at www.azure.com) is not recycled old ideas. The fabric controller - which seems like the key to this - is an amazing concept. It does to datacenters what Windows or MacOS did to PC hardware. It abstracts away the machiens and the networks and the routers so that developers don't have to think about it. Think of it this way: Windows and MacOS made it unnecessary for developers to have to worry about how to talk to the hardware, write text or graphics to a screen or create a dialog box. Why should every developer have to write the code to do those types of standard operations? Red-dog is sort of doing the same thing for distributed computing on a massive scale. The developer writes code stating what types of requirements an application has and the frabric controller does all of the undelying work to make sure the app runs as promised.

      I have no doubt that somewhere deep in Google's datacenter infrastructure they're using similar concepts but you sure don't hear much about it because it's what makes Google Google.

      Microsoft should probably consider itself lucky that there are cluless wonders like you who discount them. Lotus and WordPerfect said the same thing 20 years ago when Word and Excel first came out. The Unix crowd said the same thing when Windows NT first came out. IBM and Novell said the same thing when Exchange first came out. IBM and Oracle said the same thing when SQL Server first came out. Years later it turned out that Microsoft's technology proved to be very popular.
      marksashton
      • And you "claerly" haven't done your spelling.

        NT
        Zach S
      • Excellent post

        I agree with you, and while I'm not familiar with Red Dog it's not a new concept. HP used to have something called the UDC (Unified Data Center) that did something similar but after the merger with Compaq the new management couldn't understand the benefits, they preferred to just push hardware servers over ground breaking technologies. I know they sold a few, and I spoke to one customer who had one and they said they cut their deployment time for new apps from days to just minutes. It had a great GUI interface and you just built your environment using building blocks like Storage, Servers and network bandwidth. If it had caught on it would have put folks like me out of a job! With a US$6M starting point (back then) they didn't sell a lot. I've heard that both IBM and Sun have experimented with similar concepts, anyone know anything about that?

        IMHO, this could be the strategy that saves Microsoft from itself. They have the resources to pull this off, and with Cutler driving they have a good chance for success. Right now they are heading down a rathole with Windows that I doubt they can survive in the long term, but with a solid Cloud strategy they might just reinvent themselves in time. Of course that's just my opinion, I could be wrong.

        "You never change something by fighting the existing reality. To change something, build a new model that makes the existing model obsolete." - R. Buckminster Fuller (1895-1983)
        914four
      • You claerly haven't done your reading

        So, you can't spell... and "clearly" you can't comprehend what you read. As for popular, yes, sucked in to Bill Gates fairy tale. Reliable, "NO". Viruses, hang ups, and crashes, I Love "My" Microsoft "Windoze"
        I'm glad you do.
        Windows based O.S.s were already the logical way to go and the operating systems of their day were already Xwindows based. Billy didn't invent it as Microsoft would lead you to believe, he just purchase one, fiddled with it and named it Microsoft Windows 3.1/9x/"others"/XP/VISTA/Server 2003/2008/"7???", I think I mentioned almost all the lemons. As for other O.S.s, you think Microsoft did it all, Not just you , but all of you, wake up out of your dream world, but before you do what colour is the sky there. There are other companies that have already produced and are improving their Cloud "Environments". "NOW" Microsoft is developing their own? Sounds more like they bought some program code from some poor sap, slap a GUI around it and plaster it with Microsoft Logos, so that people like you and the writers, that can not seem write about any other operating systems, can continue to complain about the same old bug, virus infested, and memory leaking Microsoft Windows version "Whatever" and now another Microsoft slap together, instead of switching to something that works even minutely better let alone something that does. Some people see the trees for the forest, others complain they can't even see the forest.
        Cheers,
        Scoob
        scoobee@...
    • Obviously

      or it could be that MS is going to let Apple and Linux focus so much on the desktop that they'll have an entire system and strategy in place when the desktop no longer matters.

      Anyone who thinks MS will be irrelavent anytime soon really needs to pull their heads out of their arse. They have far too many insanely smart people to let that happen. Ray Ozzie will turn out to be more important 10 years from now than Bill was the first 20. That's my take.
      LiquidLearner
      • Linux focus

        Linux is already in "the cloud" as well as the desktop and the cell phone market and everywhere else you can imagine. In fact, ubuntu 9.10 will have cloud-building tools built in to make it easier to have a personal cloud, specific for your company, built. Not the least of the perks from those tools is that they are free and open source.

        Microsoft came to the party late. It gets by on the brand-name, that's it.

        With that said, I hope I'm wrong.
        tmsbrdrs
      • "too many insanely smart people"

        For once you got it right.

        I've always heard that there's an idiot in every genius. This must be proof positive.
        Ole Man
      • GM

        GM used to be the biggest auto company in the world with lots of insanely smart people. But where are they now?
        Axsimulate
  • RE: Red Dog: Can you teach old Windows hounds new tricks?

    Yes.
    Loverock Davidson
  • "... infamous ..."?

    What did he do to become notorious?

    Quoting:

    ... everyone from the infamous father of VMS and NT, David Cutler ...

    [End quote]

    Did you mean just "famous"?
    Anton Philidor
    • infamous

      Hi. No, I meant infamous. Have you ever read the book Show Stopper!? If you have a chance, read it. I think it is the best book ever written about Microsoft. And if you do read it, you'll see why I added the "in" to the "famous"... :) MJ

      Show Stopper!: The Breakneck Race to Create Windows NT and the Next Generation at Microsoft

      http://www.amazon.com/Show-Stopper-Breakneck-Generation-Microsoft/dp/0029356717

      Mary Jo Foley
      • Okay.

        But he did get the job done.

        And in the Microsoft system, which appears based on leadership and not a self-reliant group. To me, that's a weakness.

        During the second World War, US soldiers were warned to stay out of Patton's training area. The troops were disgruntled enough to shoot people on their own side in their own country. But Patton was considered one of the most effective US generals.

        Sometimes efficiency and effectiveness are the primary grounds for evaluation.
        Anton Philidor
  • What kind of an operating system?

    What is Red Dog?

    At first introduction, it's only named:

    The part of Azure which intrigued me the most was the cloud operating system, code-named ?Red Dog,? that is at its heart.

    [But slowly clues become available:]

    The findings: Cutting costs was essential. Providing greater levels of reliability was essential.

    ?If we could get the number of machines down or reduce the amount of required support staff,? that was going to be huge,? Proebstin said. The potential Red Dog customers wanted to know how long it would take for a new feature to be introduced or to address an uptick in server demand. Those were their hot buttons, he said.

    ...

    At this point, the team realized the key to the cloud was to be able to better manage the datacenter.

    ?The idea became managing a datacenter as an operating system,? Srivastava said. ?We wanted to abstract the whole thing and manage all the resources.?

    In April 2007, the team began writing code for the components it needed to run Windows Server 2008-powered datacenters: The fabric controller, a storage system, a virtualation substrate and a development environment. By November 2007, the first version of Red Dog was working.

    [End quote]

    Those are enough clues to make a good guess. Windows is safe.

    But since it's the topic of the article(s), is it possible to describe in a single sentence what Red Dog is?
    Anton Philidor
    • Twitter-Style Red Dog Definition

      My Take: The Windows Azure OS is Windows Server 2003 Core running on a customized version of the Hyper-V hypervisor, which supports VMs that run Windows Server 2003 Enterprise.

      --rj
      Roger_Jennings