Microsoft's Midori: The M# connection

Microsoft's Midori: The M# connection

Summary: There've been a couple of recent developments of note regarding Microsoft's 'Midori' skunkworks operating system project.

SHARE:

Over the past couple of weeks, there were two interesting developments around Microsoft's "Midori" operating system project.

xmarkstheMsharpspot
"X" marks the targeted M# spot, according to this illustration by Midori team member Joe Duffy.

First, I heard from two of my contacts that Midori -- Microsoft's non-Windows-based operating-system project -- moved into the Unified Operating System group under Executive Vice President Terry Myerson. (Before that, it was an incubation project, without a potential commercialization home inside the company.)

Secondly, Microsoft officials seemingly gave the green light to some of the Midori team to go public with more details about the project. Specifically, Joe Duffy, one of the Midori team members, blogged on December 27 about the language used to develop Midori, which the Midori team built alongside the OS itself.

That language, codenamed "M#," ("M Sharp")according to several different sources of mine, is an extension of Microsoft's C# language. According to a self-identified former Softie on a Reddit thread discussing Duffy's post, this new language "grew out of Sing#, the system language of Microsoft Research's Singularity OS."

And according to Duffy's post about the new language, which he calls "C# for Systems Programming," this new language may ultimately be open sourced. But before that happens, Duffy and his cohorts plan to reimplement M# on top of Microsoft's still-unfinished "Roslyn" compiler-as-a-service technology.

Midori: What we've gleaned so far

A skunkworks team inside Microsoft has been working on Midori since at least 2008 (which is the first time I blogged about the Midori codename and effort). The Midori team can trace its early roots to "Singularity" the Microsoft Research-developed microkernel-based operating system written as managed code.

Midori originally was championed by Microsoft Chief Technology Officer Eric Rudder. The Midori team consisted of a number of all-star Microsoft veterans (indluding Duffy), plus some additional high-flying developers from outside the company.

Early Midori design documents indicated that the Midori OS would be built with distributed concurrency and cloud computing in mind. Microsoft also supposedly planned to try to provide some kind of compatibility path between Windows and Midori (which isn't based on Windows at all). The early design documents also indicated that Microsoft Research's "Bartok" ahead-of-time compiler work influenced the team.

Duffy made a couple of public presentations and published papers in the ensuing years that indicated he and his colleagues were working on some kind of extensions to Microsoft's C# language. In his new blog post, Duffy elaborated on this, noting that M# (a codename he never uses himself) "should be seen (as) more of a set of extensions to C# — with minimal breaking changes — than a completely new language."

Duffy explained that the team made a conscious decision to work from C# as a base, rather than C++ -- the language a number of developers on the Windows team have favored -- because there's a substantial pool of C# talent. Duffy also said the team's goal of "minimizing complexity" resulted in a preference for C# rather than C++ as a starting point.

"Type-safety is a non-negotiable aspect of our desired language, and C# represents a pretty darn good 'modern type-safe C++' canvas on which to begin painting," Duffy blogged. He added in the comments on his blog post: "A goal was that any C# compiles in this new language, and then there are a bunch of new features that are opt-in."

A number of individuals commenting on Duffy's post (and in the aforementioned Reddit thread) wondered aloud why Microsoft simply didn't use other languages, like Go, Rust or D, rather than inventing a new C# derivative.

Aleks Bromfield, a former member of the Midori language team who joined the Google Developer Infrastructure group in July 2013, explained Microsoft's target with M#.

"We're like Rust in that we want to maximize safety and performance, and we're willing to add fancy type system features to get there," Bromfield said. "We're like Go in that we want to be simple and approachable, in much the same way that .Net attempted to simplify programming. We're like D in that we want the language to feel very familiar to devs using an existing language (C# for us, C++ for them) -- just better."

Developers theoretically could use M# to build any kind of application, including line-of-business ones, Bromfield said.

"The idea is that M# should be the lowest-level language that you'll ever need. It sits at the very bottom of the stack. But it's also safe and productive enough to be appropriate for writing higher-level systems, like Web services," Bromfield added.

What's next for Midori?

Many Microsoft watchers, including yours truly, wondered if Midori would ever exit its incubation phase. But with one-time champion Rudder moving in November to a new advanced strategy post -- plus the move of the Midori team into Myerson's OS group -- something seems to be afoot.

While Midori was in incubation, the Microsoft Research team working on the "Drawbridge" library OS managed to support Midori as a host implementation, alongside a number of other Microsoft operating-system platforms. (A library OS is a form of virtualization that seeks to replace the need for a virtual machine  to run software across disparate platforms.)

One of my contacts said Myerson's OS group is going to be determining which parts of Midori have a place in Microsoft's future operating-systems plans. Based on Duffy's post, it sounds like the M# piece of Midori will evolve throughout 2014, but it's not clear when and if it ultimately will be open-sourced.

Topics: Operating Systems, Cloud, Emerging Tech, Microsoft, Software Development, Windows

About

Mary Jo has covered the tech industry for 30 years for a variety of publications and Web sites, and is a frequent guest on radio, TV and podcasts, speaking about all things Microsoft-related. She is the author of Microsoft 2.0: How Microsoft plans to stay relevant in the post-Gates era (John Wiley & Sons, 2008).

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

41 comments
Log in or register to join the discussion
  • will M# provide the GUI

    or will the GUI be an option for the developer? or is still too early to determine if this going to be an OS , framework (ala .Net, ) or basic instruction set?
    chips1
    • M# increases C#'s scope

      From what I gather, M# would allow you to use C# for many C++ type tasks / projects, including writing drivers. The above would therefore increase the scope of tasks / projects which could be done with C#.
      P. Douglas
      • Yeah

        It would be really cool if we C# developers could start using M# to write native components and and much faster, unmanaged code. Conceivably this could unify the .Net and non-.Net worlds within Microsoft.
        christoban
    • Did you read Duffy's article?

      Or MJ's analysis above for that matter?

      The language being discussed (let's call it M# for now) is a set of extensions and modifications to the C# programming language. It's not (directly) anything to do with a GUI, OS or Framework.
      bitcrazed
      • From MJF's article, it isn't an entirely unreasonable question

        No matter how Duffy characterized it, MJF has, at least in its development history, shown that M# is apparently quite direct-line related to Midori and Singularity.

        And without being able to directly develop with it yet, and with Duffy's cryptic comments about refactoring the Framework ("noting that they needed to change pretty heavily to satisfy our goals"), it is entirely reasonable to wonder how M# would interact with presentation layer stuff, so chiding people for wondering is less that wholly appropriate.
        Mac_PC_FenceSitter
        • GUI

          Does C# provide a GUI for the user? No.
          Does C++ provide a GUI for the user? No.
          Java? JavaScript? C? Perl? Python? Prolog? Eiffel? Pascal ... No.

          M# is a programming language that Duffy himself states is "C# for Systems Programming"

          I think that clearly states that it is ... well ... a systems programming language designed and built to overcome some of the challenges presented by hard-core systems programming scenarios and is not an "OS , framework (ala .Net, ) or basic instruction set?"
          bitcrazed
  • Bye bye One Microsoft

    Just when they were merging everything into One Microsoft, they create a new OS.

    I hope they learned from the Longhorn failure.
    drfisheye
    • On the contrary

      ...by unifying C++ and C# development camps, this could be a useful move toward One Microsoft.
      christoban
  • Not sure I'd place JavaScript so high on "safety & productivity"

    Not sure I'd place JavaScript so high on "safety & productivity." Not really so safe, with most exploits using it, and not really so productive except there's really no other choice for a web language that's on all browsers without some sort of plugin.

    . . . and I'd probably place C# higher than Java on performance.
    CobraA1
    • My first thoughts

      I looked at the table and it just looked wrong! Java, safe and productive? Productive yes, but it has more holes than a spagehtti strainer! The same with JavaScript, it is an easy language to get into, but not exactly safe.

      I assume they mean "type safe", as opposed to secure...
      wright_is
      • Type safety

        No, they don't mean type safe in this case because Javascript is definitely not type safe. By safety they mean that it executes in a virtual machine so that it can't crash the process. By productive they mean that it has a garbage collector so you don't have to worry about memory management.
        mbert65
    • That isn't what is meant by safety in programming terms

      A handful of lines of C++ can BSOD your computer. There is no amount of Javascript that could, on its own, do this. That is why Safety is often related to productivity.

      If the boss is pushing you to get a product out that is only nominally ready, at least you don't have to worry about your code crashing peoples' systems. The managed code platforms (Java, .NET) and Javascript offer this.
      Mac_PC_FenceSitter
      • Not so fast...

        Any VM language can crash the process that's running it. No VM is perfect because the folks that created it aren't perfect. If there's an exploit in the VM, code running in the VM can crash the OS. The latter would be exceedingly rare but nothing is 100% and using such absolutes is risky.

        Regarding a handful of C++ lines that crash the computer, that statement too is risky. Unless those are very specific lines that exploit a known OS or driver defect, what's done in C++ shouldn't crash the OS, the OS should catch it and terminate the process.

        Regarding productivity, C++ has a lot of safety and does a great job of warning you that you are about to do something that may or may not yield the desired results. However, you can ignore/override/disable compiler warnings and continue down any path. For instance, you can declare a function pointer and a character pointer. You can initialize the character pointer to a memory location that contains your name. You can then assign the character pointer to the function pointer and invoke it. This doesn't make any sense but C++ will still do it since it's just translating to the underlying x86/x64 assembly instruction to CALL instruction. In ASM, you can call any address you want, even if it doesn't make sense. However, the OS will likely catch the mistake because the address might be in memory that is not marked as executable or it will actually begin executing code that expects the stack to be in a certain state and it isn't. Even then, the random called routine may operate on whatever stack is present and if it encounters a RET, it will return and then the program will crash because before the call a new stack frame was not setup but was likely destroyed by the random bit of code that was just executed (depending on the particular C++ call convention in use). At the very least, this would leave the code after the call with a mangled stack and ready to crash.
        robradina9
  • Are they going to grow a whole new userland and window mgr, too?

    I'm confused ... is it their intent to develop a whole new userland as well as kernel? Or will they rely on Windows compatibility for userland programs? (I'm not sure if Windows calls it "userland", btw, but the equivalent of the Linux userland, I mean).

    Maybe the thinking in that area isn't decided yet, or if decided isn't public yet, but thought I'd ask.
    daboochmeister
    • Seriously?

      M# is a programming language, based on C#, but introduces more explicit memory management (among other) features that make it more effective when building systems level code that has to be more predictable in how it allocates, manages and releases memory.

      Midori was a research project to create a whole new OS built in managed code, but this is orthogonal to the programming language used to build it, just like C/C++ is orthogonal to Windows, UNIX/Linux, etc.
      bitcrazed
      • It is a reasonable question

        Something may be orthogonal in technical terms, but often isn't in practical ones. .NET, despite Xamarin's efforts, is still strongly regarded as a Windows platform... and well, just look at Objective C. Yes, in theory, a language, one that can in theory be compiled anywhere you have the GCC.... but one heavily tied to Apple platforms.

        Let's save the chiding of people for wondering about this for a time when they can check it out for themselves, and then still don't bother.
        Mac_PC_FenceSitter
  • On DrawBridge

    Hmm - just encountering DrawBridge, I hadn't heard of it before. (see the linked article from 2011 in the above article).

    Asking for people's opinions - container-based virtualization is catching fire in the Linux world, with docker.io etc. (and of course, containers have been a standard feature of Solaris and BSD for years, though I think the growth of a whole ecosystem/PaaS approach based on them is a new thing).

    In practical terms - in terms of the net effect on system design, performance and security -- is it somewhat fair to call DrawBridge "containers for Windows"?

    Clearly, containers are somewhat more limited than DrawBridge, in that you don't get to run different OS "personalities" ... but would that matter? There are clear use cases where you need to run OSes based on different kernels (e.g., Linux and Windows) on the same host, and hypervisors support that ... but are there real uses cases for dissimilar Windows "personalities" running on a server? And if not, doesn't DrawBridge, used for all the "same" personality among all running apps, effectively become an implementation of containers? (with a little bit more duplicated in each application space than for Linux containers/Solaris zones/BSD jails).
    daboochmeister
    • There are two motivations for using VMs

      "Clearly, containers are somewhat more limited than DrawBridge, in that you don't get to run different OS 'personalities' ... but would that matter?"

      I have no inside information, but I expect this is precisely the motivation for Drawbridge. VMs are used not only for manageability, but also for compatibility, e.g. the 'Windows XP mode' in Windows 7 (which is just a VM running the full Windows XP, adding a lot of overhead). For isolation/manageability, something like LXC is sufficient, and the obvious counterpart on Windows would be virtualisation at the NT Object level (the closest thing to the Unix file tree in Windows). For compatibility, you need something more advanced, like Drawbridge.

      The primary benefit of Drawbridge, if used in a production version of Windows, would be to enable more rapid evolution of the OS. Old versions of the OS would continue to be available in library OS form, so breaking changes could be introduced without disruption. In the extreme case, users could upgrade to the new OS but run all of their existing applications with the old libraries, with minimal overhead. At the same time, all of the new features would be available to new applications. The host OS could even be transparently replaced with something entirely different (e.g. Midori).
      WilErz
      • I forgot to mention migration

        Another advantage of the library OS approach is it would allow competing operating systems to run on top of the host OS, also with minimal overhead, making migration much easier. An obvious example would be Linux, assuming Microsoft were to port it to Drawbridge. Linux servers could be transparently migrated to Windows (or Midori?), without the overhead introduced by VMs.
        WilErz
        • Its called "Microkernel"...

          Though Microsoft seems to want to rename old standard technology into new nonstandard Microsoftisms...
          jessepollard