Azure: One big, happy platform?

It has been four months since Microsoft took the official wraps off its cloud-computing initiative.  Yet still relatively little still is known about the Azure platform and plans.

The part of Azure which intrigued me the most was the cloud operating system, code-named “Red Dog,” that is at its heart. Late last month, Microsoft allowed me access to many of the principals behind Red Dog — everyone from the infamous father of VMS and NT, David Cutler, to the handful of top-dog engineers who helped design and develop the various Red Dog core components. Over the course of this week, I’m going to be publishing a post a day about Red Dog.

One platform, under Oz

While this series has focused to this point on Red Dog -- the operating system component -- of Microsoft's Azure platform, Azure also includes software and services that aren't part of the core OS.

In fact, attendees of Microsoft's Professional Developers Conference held in October 2008 might remember this diagram:

One big, happy cloud platform. Right?

In reality, while Azure looks like a cohesive platform now, just a year or two ago, different teams inside the company were all working away in their own silos on the early iterations of their  various cloud-computing schemes. The Live Mesh team was thinking about how to create a consumer-focused collaboration service. The Connected Systems Division (CSD) was thinking about how to get its middleware out, in service form, to business users. And the core Red Dog team was trying to figure out a game plan by surveying existing services teams at the company.

Azure and Live Mesh "were conceived around the same time," said Abhay Parasnis, General Manager for Live Mesh.

"When we started Mesh, from Day 1, we knew an OS tier had to be done," Parasnis said, but that OS was not part of the Mesh platform. Red Dog ultimately became that tier, with Live Mesh evolving into the consumer-facing "scenario" that would demonstrate how/why a cloud-computing platform was relevant to end users.

Meanwhile, on the enterprise side of the Microsoft house, CSD was thinking along the same lines.

"The Red Dog team was doing a set of incubations around Live Services and we (CSD) were doing stuff around developer services," recalled Technical Fellow John Shewchuk, who was instrumental in developing the "Zurich" services that became part of the Azure platform. "(Chief Software Architect) Ray (Ozzie) didn't know about us early on in the Red Dog process."

But once Ozzie and the Red Dog team learned what CSD was doing, merging the different platforms seemed the way to go, Shewchuck said.

"We ended up being one of the first teams to port our services to Azure," Shewchuck said. "We are pickiing off our services and bringing them over (to Azure), one by one."

To get the various teams with a stake in the cloud on the same page, Ozzie instigated a couple of different cross-divisional meetings.

(In spite of many outsiders' view that Microsoft coordinates its attacks on markets across its different business units, at least up until now, that hasn't been how things work. Managers of different business units at the company often had no idea what other units were doing -- except at the highest level.)

Parasnis, Shewchuck, members of the Red Dog operating system and developer platform teams started meeting on a fairly regular basis.

"The idea is every senior leader now knows what is going on in the other (parts of the Azure) world," Parasnis said. The participants discuss things like "Does a developer have to sign up three times, or can they get a single ID to work across all of the platforms?" Parasnis explained.

At the same time, members of different parts of the Azure platform also are working one-on-one with their colleagues to address particular problems.

"Our engineering efforts are fully aligned with Red Dog now," said Shewchuck. "We expect them (Red Dog) to be available with a fully integrated developer experience" upon which CSD and its customers can count when working with .Net Services, SQL Data Services and other components.

[Learning lessons -- in real time] -->

Lessons learned (in real time)

In spite of the growing Microsoft cross-divisional ties, Shewchuck emphasized that Azure isn't going to be a closed cloud environment.

"This isn't going to be a set of Microsoft-only services. It's not a closed environment," Shewchuck said. Java, PHP and Ruby developers will be supported from "the get-go," he said.

(I'm hearing rumblings that Microsoft might use its Mix '09 conference to tout how the Azure development environment will work with non-Microsoft tools. Stay tuned....)

In addition to the importance of being open to non-.Net-focused developers, what else is Azure teaching Microsoft?

"Feedback is now immediate," Shewchuck said. By last October, the BizTalk services team had issued 13 mini-releases of the main piece of the .Net Services platform over the past two years, he said.

"In essence, our monthly builds were going out to the community," Shewchuck said. Originally, that idea "was viewed with a lot of skepticism inside STB (Microsoft's Server and Tools Business)," he said. "Many people viewed a project operating like that as a toy." But they're coming around, he said.

Another learning -- one which the Red Dog team also absorbed -- the wheel doesn't need to keep being reinvented.

".Net Services was built by many different organizations," Shewchuck said. "We didn't build everything from the ground up." The Federated ID/Geneva teams, the virtualization folks, the messaging team all contributed pieces, he said.

"We had to grow this new muscle, our own rapid development structure," he said. "We needed to change our testing techniques. And we needed to learn about running services at scale."

Parasnis agreed. "It's not about taking the classical approach to building a platform. We needed to be iterative and change the platform as you get the feedback. It was a fundamental skill set we needed to acquire."

(My Red Dog series ends tomorrow. No more hints on what's up next. You'll just have to come back.)