Must Read: Mint 15: Today's best Linux desktop (Review)

Topic: Operating Systems

'Unix beats Windows' - says Microsoft!

Summary: So why is this interesting? because their test methods reflect Windows internals, not Unix kerneldesign. There are better, faster, ways of doing these thingsin Unix, but these guys - among the best and brightest programmers working at Microsoft-either didn't know or didn't care.

Paul Murphy

By Paul Murphy for Managing L'unix |

Ok, that headline may be a bit overblown - but Microsoft Research has released part of a report on the "Singularity" kernel they've been working on as part of their planned shift to network computing. The report includes some performance comparisons that show Singularity beating everything else on a 1.8Ghz AMD Athlon-based machine.

What's noteworthy about it is that Microsoft compared Singularity to FreeBSD and Linux as well as Windows/XP - and almost every result shows Windows losing to the two Unix variants.

For example, they show the number of CPU cycles needed to "create and start a process" as 1,032,000 for FreeBSD, 719,000 for Linux, and 5,376,000 for Windows/XP. Similarly they provide four graphs comparing raw disk I/O and show the Unix variants beating Windows/XP in three (and a half) of the four cases.

Oddly, however, it's the cases in which they report Windows/XP as beating Unix that are the most interesting. There are three examples of this: one in which they count the CPU cycles needed for a "thread yield" as 911 for FreeBSD, 906 for Linux, and 753 for Windows XP; one in which they count CPU cycles for a "2 thread wait-set ping pong" as 4,707 for FreeBSD, 4,041 for Linux, and 1,658 for Windows/XP; and, one in which they report that "for the sequential read operations, Windows XP performed significantly better than the other systems for block sizes less than 8 kilobytes."

So how did they get these results?

 

The sequential tests read or wrote 512MB of data from the same portion of the hard disk. The random read and write tests performed 1000 operations on the same sequences of blocks on the disk. The tests were single threaded and performed synchronous raw I/O. Each test was run seven times and the results averaged.

umm...

 

The Unix thread tests ran on user-space scheduled pthreads. Kernel scheduled threads performed significantly worse. The "wait-set ping pong" test measured the cost of switching between two threads in the same process through a synchronization object. The "2 message ping pong" measured the cost of sending a 1-byte message from one process to another and then back to the original process. On Unix, we used sockets, on Windows, a named pipe, and on Singularity, a channel.

So why is this interesting? Because their test methods reflect Windows internals, not Unix kernel design. There are better, faster, ways of doing these things in Unix, but these guys - among the best and brightest programmers working at Microsoft- either didn't know or didn't care.

And if they're the best and brightest, what do you think happens when the average Microsoft programming whiz gets asked to program for Linux?

Topic: Operating Systems

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

111 comments
Log in or register to join the discussion

  • More evidence

    That you cannot compare Windows and *nix side-by-side and yield meaningful results. It doesn't matter who is doing the comparison, there are enough core differences in the design and implementation of both systems that the majority of comparisons are meaningless.
    Real World
    Reply Vote

    • You really want to make these "big" statements?

      Can you also claim that, for instance, 2 and 4 stroke engines are not comparable because their internals are different?

      I can compare the PERFORMANCE of an OS to that of another, using unbiased benchmars and experimental conditions. No?

      Beacuse what matters is the performance EXPERIENCED (as view) from the application perspective. No?

      It is fairly "simple" to compile the same benchmarks on the various OS platforms and then carry out the experiments.

      I can launch multithreaded code and test the cost syncr. and scheduling of the various threads. Semaphores, mutexlocks, and other IPC mechanisms are found on ALL systems. Why couldn't someone measure these times?

      I can do memory, network I/O, disk I/O and file I/O testing with different access patterns measured under identical conditions and tabulate the results. This is NOT nuclear science but you NEED to jave intimate understanding of internals and basic statistics so that you will not taint the results inadvertandly (or intentionally like someOUne else I know)

      WHY again the comparisons are meaningless?

      You are not a developer that's why you cannot see the connection between the cost of various OS services to application performance.

      Visit the http://www.spec.org to get an IDEA of benchmarking. This is by NO means an exhaustive performance comparison of platforms but it is a start.

      --m
      michael_t
      Reply Vote

      • Benchmarks are

        subjective, a point that Paul alluded to in the blog above. I don't expect you to understand that, however. You're the one that required 6 posts to finally grasp that Windows XP has a system restore utility, and Windows 2000 doesn't, right? And you may want to devote some time to working on your English.
        Real World
        Reply Vote

    • You really want to make these "big" statements?

      Can you also claim that, for instance, 2 and 4 stroke engines are not comparable because their internals are different?

      I can compare the PERFORMANCE of an OS to that of another, using unbiased benchmars and experimental conditions. No?

      Beacuse what matters is the performance EXPERIENCED (as view) from the application perspective. No?

      It is fairly "simple" to compile the same benchmarks on the various OS platforms and then carry out the experiments.

      I can launch multithreaded code and test the cost syncr. and scheduling of the various threads. Semaphores, mutexlocks, and other IPC mechanisms are found on ALL systems. Why couldn't someone measure these times?

      I can do memory, network I/O, disk I/O and file I/O testing with different access patterns measured under identical conditions and tabulate the results. This is NOT nuclear science but you NEED to jave intimate understanding of internals and basic statistics so that you will not taint the results inadvertandly (or intentionally like someOUne else I know)

      WHY again the comparisons are meaningless?

      You are not a developer that's why you cannot see the connection between the cost of various OS services to application performance.

      Visit the http://www.spec.org to get an IDEA of benchmarking. This is by NO means an exhaustive performance comparison of platforms but it is a start.

      --m
      michael_t
      Reply Vote

    • Definately agree

      This is even more true than the fanboys of all sides in the Mac OSX vs. Windows vs. FreeBSD vs. Linux debate. At least some of those operating systems are slightly similar in implmentation. Singularity is a micro-kernel and of a completely different design (in managed code no less, what a feat). But again, performance was not the goal of the group, merely a happy side effect of their design.
      Someguy2
      Reply Vote

    • more evdence

      except wehn the comparison ius deliberately weighted towards the losing side. then you begin to wonder
      giskard
      Reply Vote

  • Singularity Kernel Looks Interesting

    I don't usually compliment MS, but the singularity kernel, from what I read in the link to the paper, looks quite good. I wonder what Andrew Tanenbaum would make of it :-)

    <sarcasm>But how will IE ever be able to operate if it's not part of the OS?</sarcasm>

    A good read.

    The Banjo
    BanjoPaterson
    Reply Vote

    • Agreed - very interesting indeed

      Particularly since many of their security ideas
      (unlike factotum) are not well designed for portability. In fact, from what they've said, I'd guess this thing would work more or less ok on AMD, work well on PowerPC and SPARC, and pretty much dog it on products like celeron and the P4.
      murph_z
      Reply Vote

      • Well well

        Saw that.
        Too funny.
        D T Schmitz
        Reply Vote

      • funny how you omit and misrepresent facts

        how come you've not mentioned Pentium M. Would that kick ass when compared to PowerPC and SPARC.

        Also why is it that x86 from ADM is just OK. Are you comparing 32 bit x86 to 64 bit PPC, Sparc for data intensive applications to say that AMD would work more or less ok.
        zzz1234567890
        Reply Vote

    • More on Singularity

      There's a great video where the two main designers of Singularity talk about the project in depth, what it means, and why they did it.

      http://channel9.msdn.com/Showpost.aspx?postid=68302
      Yensi717
      Reply Vote

  • And how often...

    ...does the "average Microsoft programming whiz" get asked to program for Linux? Conversely, how often does the "average Linux programmer" get asked to program for Windows? And when he does, does he make an effort to do things the "Windows way", or does he try to force-fit Unix paradigms onto Windows, then complain about Windows' inferiority?

    Seems to me it can (and does) work both ways.

    Carl Rapson
    rapson
    Reply Vote

    • Good point(nt)

      nt
      IT Scion
      Reply Vote

    • Sure .. but we're right

      In all seriousness, most Unix programmers have some theoretical understanding of what they're working with - and are correspondingly more likely to look beyond their immediate experience for ideas and methods.
      murph_z
      Reply Vote

      • Paul...

        ......I really enjoy your blogs, and I frequently find something in the to make me think about different aspects of things. But terms like "most Unix programmers" and "average Windows programming whiz" indicate a tendency to generalize which is not becoming in a professional observer. You seem to fall into the same stereotyping trap that most anti-Microsoft pundits (and talkback fanboys) do. On what do you base this observation? Have you personally met most Unix (or Windows) programmers in the world, to be able to make such generalizations?

        The fact that you recede it with "In all seriousness..." is even more distressing.

        Carl Rapson
        rapson
        Reply Vote

        • Every single stratum of them..

          Of course not, but your comment makes me wonder about something else: is sampling theory applicable to experience?

          i.e. It's not possible to meet every Windows programmer, but I've certainly worked with hundreds and since sampling theory supports the notion that polling them at random would produce results applicable to all of them as a population...

          Now, you could argue that my experience doesn't reflect a random hike through their ranks - but how would you support that? or, put another way: I can't say all men are taller than all women, but I certainly can say that the average man is taller than the average woman. And if sampling theory allows that, why not other experience?
          murph_z
          Reply Vote

          • Not in my opinion

            "...is sampling theory applicable to experience?"

            I don't think so, and for precisely the reason you mention: your experience isn't sufficiently representative of the entire population. I don't think anyone can legitimately claim that their personal experience constitutes a sufficiently large and random sampling of the entire population to be able to draw generalizations (especially considering the size of the Windows programmer population). COnsidering that your field of expertise is Unix, I'd expect that your experience "sampling" is biased toward Unix and away from Windows, just as I'd expect a Windows programmer's would be biased against Unix. That's why I don't buy such generalizations from either side.

            I don't think your height analogy isn't quite an exact match, either. We can generalize that "most men are taller than most women" because we can obtain separate, scientific evidence of such, outside of our own personal experience. We can find genetic evidence of the height difference between men and women. We can see, through the various media, a vastly larger sampling of the entire human population than we can possible interact with. How many Windows programmers are there toiling away at home and within small-to-large corporations that no pundit ever "sees"?

            Most of the "evidence" that Unix is "better" than Windows consists of an innate bias against Windows and a refusal to recognize the differences in the platforms. Windows programmers who attempt to work in Linux using Windows paradigms are called incompetent, but Linux programmers who attempt to work in Windows using Unix paradigms are called superior. When Unix is held up as not only the ideal but the ultimate OS, how can Windows not suffer in comparision? The deck is stacked before you begin.

            Carl Rapson
            rapson
            Reply Vote

          • Agreed - but I wasn't being totally serious

            In fact, I was spoofing the argument because I don't actually disagree with you on this.

            And, by the way, your comment:
            " Windows programmers who attempt to work in Linux using Windows paradigms are called incompetent, but Linux programmers who attempt to work in Windows using Unix paradigms are called superior"

            leads me to a request: wanna proof read/argue with a 50 page doc for me? if so, please get in touch.
            murph_z
            Reply Vote

          • OT - proof read

            Paul, should I contact you here or on your personal web site?

            Carl Rapson
            rapson
            Reply Vote

      • Sorry, they don't

        Unix programmers are like everyone else when it comes to writing for an unfamiliar platform. For example, I remember some work done by a Unix house to run on Vax VMS. Because both systems allowed processes to "fork" new processes (something Unix does easily), they tended to use this a lot, though forking on VMS is much more burdensome and you would do better to use threads, as for Windows (which inherited some VMS architecture).

        Developers don't spend a lot of time understanding system internals. They read the manuals to find a function that does what they think they need to get the job done, typically by looking for one with the same name as they are familiar with.

        And that is often not the best answer.
        A.Sinic
        Reply Vote
CNET Join | Log In | Privacy | Cookies Close