The performance debate: Linux vs Windows

The performance debate: Linux vs Windows

Summary: My January 13th blog To beat Linux: scam the customer drew quite a lot of talkback comments. One of the themes there was fired off by a comment from No-Ax-To-Grind to the effect that the Linux community is somehow afraid of a direct performance comparison against Microsoft.

SHARE:
147

My January 13th blog To beat Linux: scam the customer drew quite a lot of talkback comments. One of the themes there was fired off by a comment from No-Ax-To-Grind to the effect that the Linux community is somehow afraid of a direct performance comparison against Microsoft.

I'm pretty sure that Microsoft's challenge was little more than a publicity stunt The near total absence of truly comparable information can't be a coincidence. carefully structured to ensure no one could take them up on it, but the fundamental question is interesting: for some range of tasks, which is really faster given identical hardware and skills?

Running a publishable benchmark isn't cheap and, of course, there are both sales and ego reasons for companies to ensure that the results they publish support their sales. In most cases that means they try to get the best possible results -i.e. you're not going to see IBM hire just any idiot off the street to set up and run their TPC/C Linux benchmark: they're going to put some real expertise into getting exactly the results they want, and so will Microsoft and HP and everybody else who gets into this game.

As a result the public benchmarks offered by TPC and SPEC may be good places to look for an answer. Unfortunately, there are few pairs of comparable results and most of the ones that do exist reflect HP's attempts to convince people that the Itanium is worth looking at.

The TPC/H series, for example, shows an that HP Itanium2 server running Windows with SQL-Server on 32 CPUs reached a score of 30,956 on the 3000GB test while essentially the same machine, but with 64 of the same CPUs and running Oracle under HP-UX, claimed a score of 71,847 - 31% more than the doubling you'd expect if scaling were linear and the performance per CPU for the Windows server combination equaled that of Oracle under HP-UX.

You get somewhat better comparability under the TPC/C V5 benchmark. Here, for example, an HP 64 CPU Itanium2 running at 1.5Ghz was used under both software scenarios. With Microsoft SQL Server 2000 Enterprise Ed. 64-bit, running under Microsoft Windows Server 2003 Datacenter Edition 64-bit and the Microsoft COM+ Transaction Monitor, this machine obtained a throughput score of 786,646. With Oracle Database 10g Enterprise Edition running under HP UX 11.iv2 64-Bit Base OS and the BEA Tuxedo 8.0 Transaction Monitor, it reached 1,008,144 to suggest about a 30% performance advantage to Unix.

A pair of 32CPU Itanium2 based NEC machines show the same pattern. With Microsoft SQL Server 2000 Enterprise Ed. 64-bit, Microsoft Windows Server 2003 Datacenter Edition 64-bit, and Microsoft COM+ the test system achieved a score of 577,531. With Oracle Database 10g Enterprise Edition, and BEA Tuxedo 8.1 on SUSE LINUX Enterprise Server 9 it reached 683,575 - an 18% advantage for Unix.

Beyond that, however, there are a pair of two way comparisons in which both sides use the same database and the same Xeon hardware to produce a pure OS to OS comparison that Linux wins - but there's a strange gotcha hiding in the weeds.

The first of these pairings offers a direct comparison between IBM DB2 UDB 8.1 running under Microsoft Windows Server 2003 Enterprise Edition and the same database product running under SUSE LINUX Enterprise Server 9. The Unix result, 5090 on TPC/H at 300GB, is trivially better (about 1.7%) than the Windows result (5,003).

Similarly, a TPC/C V5 comparison between an HP Proliant ML350-T03-X2.8/533 running IBM DB2 UDB Express Edition v8.1 with Microsoft COM+ on Microsoft Windows Server 2003 Standard Edition and the same hardware, database, and monitor but using SUSE LINUX Enterprise Server 9, produced a 1.8% victory for Linux: 18,661 to 18,318.

However close inspection of the detailed reports suggests something very odd: the machine's controller and RAM specifications were the same in both cases - and that's not what you would expect.

Because of the waterfall object control heirarchy built into Windows 5.X (and, indirectly, the Intel Itanium), you maximize performance by using one controller (and one NIC) per CPU. Linux, in contrast, is a true SMP system so you maximize performance by reducing device interupts and loading up on memory for your database cache instead. That's why if you look at Linux benchmark configurations you'll typically see 0.5 controllers or fewer per CPU while Windows benchmarks almost always have 1:1 CPU/Controller ratios.

Since RAM is actually cheaper and faster than controllers and disks, keeping machine configurations essentially identical artificially slows Linux and raises its cost relative to Windows. Cost per transaction is, of course, one of the TPC's critical metrics, and Linux did "win" both comparisons; but it won by a much smaller margin than it should have.

Overall, however, there seem to be two trends visible in both the TPC and SPEC benchmarks:

 

  1. where the hardware is roughly comparable Unix, whether Linux or anything else, wins - usually by a wide margin; and,

     

  2. recent Microsoft benchmarks, to the extent that these are published, tend to have been done by third parties - meaning that Microsoft has apparently stopped participating in these kinds of comparisons.

So there's an interesting twist to the answer for the original question: Although the evidence we do have heavily favors Unix, there isn't enough information to draw firm conclusions. On the other hand the near total absence of truly comparable information can't be a coincidence. Combine that with Microsoft's general refusal to play in the more complex benchmarks and we can guess who's afraid of who - and it isn't Solaris, Linux, or any other Unix.

Topic: Operating Systems

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

147 comments
Log in or register to join the discussion
  • No-ax is a shill

    I believe a paid shill, probably buried in one of MSFT's PR contracts and gets more attention than he deserves for technical issues. At least George is honest about his bias and background.

    A lot of these technical benchmarks amount to nothing more than how many angels can dance on a memory stick. I'm not disputing your analysis, which was pretty fair. I'm saying you can take any platform and make the numbers come out in its favor. Most of the metrics are meaningless in day to day business.

    Like George and his load times. Sound and fury signifying nothing. Yes, Word loads faster. Happy now, George? It's meaningless. OpenOffice will do the job capably for 99.99% of all the work most of my customers do on a daily basis. The fact it takes a few seconds longer to load the first time is immaterial. It does a capable job and it can be downloaded for free, saving companies millions in license fees. There are very few compelling business cases for picking Office over OpenOffice. Every so often one pops up, but it's the rare exception. The vast majority of the time user preference is dictated by emotion rather than requirements.

    On the web side of the house I support MSFT stacks and LAMP applications. Let me promise you that, on average, the MSFT stack A) Costs more both initially and downstream and B) Requires more operator intervention to keep working right. I'd also argue that development of a web app in .NET, again on average, takes longer to build and more effort to maintain than a similar application written in PHP. That's not entirely a fair comparison but my hands on experience in both worlds tells me there is nothing "rapid" about ASP.NET.

    I do like SQL Server. I'd say that was one of MSFT's better products from a reliability perspective. But they keep boosting the pricing on it to the point you really need a business case to justify not using MySQL or PostgreSQL. Again I deal mainly with web apps.
    Chad_z
    • Don't curse the messenger

      I don't think Don/No_Ax is a paid shill. While I am not an M$ fan I do think they are smart enough to spend any possible shilling budget on higher quality posters.

      While implicitly denigrating him above I also feel that his contributions should stand on their own merits or lack thereof. To read him is to understand him as a commentator and as a person.

      The important issue here is that he has triggered the column we are responding to now. For that we should thank him even if the results of the comparison aren't irrefutable evidence either way. The net effect is that we have something new to think about and comment on other than each other.
      Still Lynn
  • Slightly misleading math error...

    "...Windows with SQL-Server on 32 CPUs reached a score of 30,956 on the 3000GB test while essentially the same machine, but with 64 of the same CPUs and running Oracle under HP-UX, claimed a score of 71,847 - 31% more than the doubling you'd expect if scaling were linear..."

    The *difference* between the two measures is 31% more than you'd expect if they scaled identically. When you compare the HP-UX score with a doubled Windows score, HP-UX scores 16% better.

    Same winner. Different spread.
    johnay
  • Debate away

    The times are a changing.

    All it takes is a little 'intestinal fortitude' and the more users sample Linux (Ubuntu, Linspire some of the more 'user-friendly' Distros, to name a few), the more they will cultivate an intelligence that Windows is 'inferior'.

    It'll will be a 'tough' year for Windows with continuing nibbling around the edges of market share by Linux and Apple!

    Thanks again for another 'quality' blog Murph.
    D T Schmitz
    • Re: Debate away

      "(Ubuntu, Linspire some of the more 'user-friendly' Distros, to name a few)"

      Mepis is sucking wind lately, Ubuntu didn't thrill me too much (Gnomes are for Trolls, etc.) and Linspire is an obvious opportunistic distro on the backs of the open-source community. Now, PCLinuxOS 0.92 (the nvidia7676 version), even with the slighty buggy sound server (there are ways around it) is working fantastic on my machine. Everything is here, Flash, Java, Open Office, practically every media player (open-source) under the sun...the list goes on. I have some video files that refuse to open with the latest players and codecs in Windohs but low and behold, no problem in Linux! >:-]
      nix_os_fan
      • Thru a big straw

        Interresting. True.

        But the ones I mentioned are 'potentially' (there are others I am sure) good for newcomers who need 'trainer wheels'.

        I am partial to SuSE 10.0.
        Thanks nix! :)
        D T Schmitz
        • You are wellcome...

          PCLinuxOS is maturing fast. It is very pretty with the experience theme and it works. It would appear that either build your own distro from scratch (which I am far too busy and lazy) or the ones that have tweaked packages (i.e. blahblah-tex, Tex is the main author of PCLinuxOS) have the least bugs. I tried Debian Sarge 3.1r1. Eek. The Debian community gets an A for effort but geeze, are they slooooooooooow. As for SuSE, the last version that worked well on my machine was 9.something. Maybe I need a new machine so the bugs show up faster, heh. >;-]
          nix_os_fan
    • Sun

      Oh sorry Murh, let's not leave SUN out (Solaris)
      D T Schmitz
  • Cost per transaction is one thing...

    ...but I noticed that in the previous article, where the author talks about how loyal Microsoft people came in to save the day, the discussion focused mainly on the cost of moving the data to another system. In other words, MS was able to save a sale by saying that if you want to move away from us, it's going to cost you.

    This, in my mind is the clearest sign that Microsoft owns your data once you start using their software, their development tools and environments, and so on.

    We all know this as Vendor Lock-in. So if MS is *so* great, why do they rely upon vendor lock-in to retain customers?

    Even if the performance gains weren't that great against MS, I wouldn't want my data locked into a proprietary format from MS. Sure, I could migrate, but then MS will always point out the migration cost to an executive who is going to look at the bottom line for the short term, rather than the technical merits of the migration.

    What should be included in TCO is migration costs if for one reason or another, you don't like the vendor you're with. And if you're working with completely documented formats and transferrable development languages, then you could more readily establish a metric for migration.

    But MS will never completely document their formats, unless of course, you pay a substantial fee and sign an onerous NDA.

    This is one argument I sincerely believe that no MS apologist could overcome. Ever.

    Scott
    Scottman_z
    • Data migration costs

      [i]"..discussion focused mainly on the cost of moving the data to another system..."[/i]

      Data migration costs are nearly always unexpectedly high when moving between any two systems, that is not peculiar to any single vendor.

      There are two main issues: data integrity and differing data models. Every business will tell you that their data is clean, but as soon as you try to migrate it you discover all the hidden nasties that must be fixed. Migrating data between different data models can be a serious challenge, particularly when relationships are based on stored procedures or similar nastiness (and SQL Server is certainly not alone in that capability).
      Fred Fredrickson
  • I want virus free and crash proof...

    reliability. I could care less how fast Windohs is. >:-P

    P.S. YEAH, YEAH, I KNOW LINUX IS NOT 100% VIRUS FREE, so don't even go 'there'. >:-P
    nix_os_fan
    • A rule for understanding posts.

      Anyone who says Windows is virus and spyware riddled and frequently unreliable has shown only that he:

      - does not know how to set up XP
      - does not know how to use XP
      - goes to websites where he's taking chances.

      A post making such criticisms is saying, "I do not know how to use Windows, so please help me."

      If someone cannot use Windows, hard to say what software would be in his purview.
      Anton Philidor
      • Yo Anton, I wouldn't touch...

        XP with your d**k if Billy himself was holding it. The serious Microspyware starts in Win2k sp3. 2k sp2 is as high as I go with any Micro?r@p. By the time that becomes obsolete, Linux will be putting your friend out of business. I already boot into PCLinuxOS 0.92nvidia7676 99% of the time. You Gates lovers are pathetic sheep. Wake and smell the Java, Micro?r@p will proprietary itself right out of business. >:-P
        nix_os_fan
        • Have you ever stopped to think...

          ... and then forgotten to start again?

          It seems to me that people on both sides like to make points by name calling (Microsucks, Linsux, etc.)

          Have you ever stopped to think that making a point in this way almost completely negates anything you are saying? You will in no way succeed to sway anyone's opinion by mudslinging.

          Next time, try to make your point intelligently, or just don't bother.
          AckItsMe2
          • Agree 100% n/t

            .
            mosborne
        • Yo nix

          Keep it clean? k? k!
          D T Schmitz
      • So Anton

        How did you make out with that dual-boot install 'eval' of SuSE 10.0? ;)

        Best!
        D T Schmitz
      • Then why is it that in the enterprise...

        our windows desktops cost a fortune to protect and we're always
        cleaning them?

        "- does not know how to set up XP
        - does not know how to use XP
        - goes to websites where he's taking chances."

        Where is this imaginary managed windows desktop that isn't
        affected by malware? Even Redmond loses machines from time
        to time so it would be helpful if you provide the information to
        them.

        Why do you apologise for MS here? Those of you that have paid
        for windows should be screaming for it to be fixed.
        Richard Flude
        • I accept your challenge

          [i]Where is this imaginary managed windows desktop that isn't affected by malware?[/i]

          I happen to be on my wife's computer right now: 1.
          My computer: 2.
          All of the computers at the office: 2,002

          Shall I go on?

          [i]Why do you apologise for MS here? Those of you that have paid for windows should be screaming for it to be fixed.[/i]

          I don't know, do you scream at Apple for them to fix OSX?

          http://www.zdnet.com.au/news/security/soa/Ancient_flaws_leave_OS_X_vulnerable_/0,2000061744,39234678,00.htm

          How about January's nasty, nasty remote code execution flaw? Hope you had a chance to patch your machine before some black hat took over your computer! What was that about screaming? Yeah, thought so.
          NonZealot
      • *Anyone* fits this rule? Maybe not.

        "Anyone who says Windows is virus and spyware riddled and frequently unreliable has shown only that he:

        - does not know how to set up XP
        - does not know how to use XP
        - goes to websites where he's taking chances.

        A post making such criticisms is saying, "I do not know how to use Windows, so please help me.""

        The 150,000 known viruses, trojans, malwarez, etc. don't diminish your argument that attacks are preventable by appropriate use of countermeasures. They should raise a concern that there are many millions of folks who fit your rule and do not know how to use and set up XP and may go to sites that aren't guaranteed safe. Perhaps there are also those who know better but are just careless. You may want to adjust your rule to account for the heedless if you consider them a significant percentage of the target definition.

        The complaint about insecurity and instability in XP might also not be based on the poster's personal shortcomings but on the observation that these things actually exist in the world.

        My brother makes good money cleaning, fixing and configuring peoples XP systems and installing antivirus, malware blocking and malware removal tools on their computers. So the fact of him complaining about these issues isn't due to a lack of skill and perspective on his part. Perhaps he is the exception that proves the rule, but he obviously doesn't fit within the confines of the rule.

        "If someone cannot use Windows, hard to say what software would be in his purview."

        No suggestions for what these millions and millions of users might need since XP exceeds their skills or diligence to use correctly? You may want to consult with someone with a wider knowledge of operating systems to discover what is available to and appropriate for the XP challenged if you are actually looking to discover options for them. And look for a list of guaranteed safe websites to add to your list of recommendations.

        Perhaps these folks should stop using their computers in order halt the feeding frenzy by the predators lurking in the XP pool. Let's just rule them out of the computing game since they don't measure up. Now who will we get to enforce the rule? Got your striped shirt and whistle handy?
        Still Lynn