Flaw counting comparisons useful but fall short of true picture

Flaw counting comparisons useful but fall short of true picture

Summary: The Windows vs Linux security report card that I wrote about from TechEd two weeks ago is officially out and Microsoft has stepped up its PR campaign to argue that Windows Vista has a "lower vulnerability fix and disclosure rate" than competitive Linux distributions.


The Windows vs Linux security report card that I wrote about from TechEd two weeks ago is officially out and Microsoft has stepped up its PR campaign to argue that Windows Vista has a "lower vulnerability fix and disclosure rate" than competitive Linux distributions.

Jones released the study (download PDF) and posted a primer with details on the methodology used to compare vulnerabilities disclosed and fixed in the first six-month period after a product ships.

In all four cases studied for the 6 month period after ship, Windows Vista appears to have a lower vulnerability fix and disclosure rate than the other products analyzed, including the reduced Linux installations. This affirms the early results that we found after 90 days and provides a supporting indicator that the Microsoft Security Development Lifecycle process and heightened focus on security is having a positive impact on Microsoft Windows in terms of fewer vulnerabilities.

He also, for the first time, broke out "high severity" vulnerabilities in the comparison and again Jones found that Windows Vista and even Windows XP fared better than Linux distribution workstations.

The controversial studies have been dismissed as biased propaganda -- see Talkback comments here and here -- from Redmond (Jones is security strategy director in Microsoft’s Trustworthy Computing group) but in my mind it's a useful attempt to dig into the publicly available numbers to find a measurement.

The problem I have with Jones is that his flaw counting ignores silently fixed vulnerabilities and makes assumptions on security based only on publicly documented vulnerabilities.

As a policy, Microsoft routinely ships silent fixes within its security bulletins if flaws are discovered internally. These are never assigned CVE numbers and will never appear in these comparison reports from Jones.

When I asked Jones and other Microsoft security staffers about ignoring silent fixes in these reports -- which could significantly increase the Windows flaw count -- they argued that everyone (including Linux distributions) issues patches with silent fixes. Additionally, Jones claimed that vulnerabilities discovered and fixed without help from external researchers do not put anyone at risk since they are only found internally.

This argument ignores the dramatic rise in zero-day attacks that use undocumented flaws/exploits to target .gov, .mil and other business networks. Try telling an enterprise that's been hit with a zero-day that his loss is less important because it's not a widespread risk, you just might get a punch in the nose.

So, while Jones' reports make for good discussion fodder, take them with a grain of salt. Hey, even Jones admits that he's biased.

Topics: Windows, Linux, Microsoft, Open Source, Security

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • It simply doesn't matter. Better security

    in all the OSs is a very good thing for all users. Anyway you cut it, Microsoft has really stepped up their game in this area and as the dominant OS thats good for everyone.
    • i hate myself for this...

      but I agree with you for once, yes probably will be a one time thing per year. They are stepping up, but they way they did the silent patches is very shady to say the least.
      • Not really

        Fixing holes silently and not alerting the hackers they exist is not an issue for me. In fact most of the software I use has fixes of one form or another in its updates. Like I said, no biggie...
        • If their issue is not wanting to alert hackers

          then why not just announce that they've fixed an undisclosed bug in XYZ program? That way the people who run XYZ understand the urgency in testing and applying the patch, and MS's bug stats are more representative of the real picture.

          BTW, I do agree that MS has stepped up their game since XPSP2 as far as improving security, and I do wish them well in their continued efforts. Everybody wins when MS products are more secure, even ABMers.
          Michael Kelly
          • No argument from me really, but...

            I am sure that balancing how much to disclose is a constant issue for everyone in the software game. When I think about the code our company writes I too am careful in describing how it works as that is the source of our bread and butter. Yes, we are one of the "evil" companies making (GASP) money from the sales of our code. <g>

            If I were to start describing say a dozen security issues it wouldn't take long before you start to get a good idea of how the code works. Like I said, its tough to balance all the needs for all concerned.
          • You don't even have to get that detailed

            If there's a flaw in one of your programs, just say it's an undisclosed flaw in that program. The other day I made an example that if Windows Server has an FTP flaw, they should say there's an FTP flaw. Saying that does not give away the family jewels, because everybody and their uncle knows Windows has that capability.

            Also since you are not a Microsoft you have the luxury of having a more intimate relationship with your clients. So if one of them has a bad reaction to one of your patches (like say their AV software goes south), I'm sure you have the personnel to deal with both the technical issue and calming down the irate customer. So if you choose not to release the nitty-gritty details of the patches, as long as you recommend that they get installed immediately and are willing to deal with any potential fallout should something go wrong, and as long as you don't falsely advertise your product's security record (fixing flaws does NOT imply a poor security record, on the contrary it shows you are mindful of security), then your customers should be satisfied.

            And no, it is not a sin to earn a buck, nor is it a sin to use a proprietary model. But if you do go with a proprietary model, you better be willing to take all the implied responsibilities that come along with it. It's really when proprietary companies fall short of customer expectations that people start clamoring for open source. Example: there is no 64-bit Flash player, so there is a lot of public support for Gnash. There was even more support for it before the newest Linux 32-bit version came out, but since then some of that support has dwindled. Expect more support to dwindle if Adobe releases a stable 64-bit player.
            Michael Kelly
          • Like I said, no real argtument from me. (nt)

        • Yeah...

          FYI, a lot of organizations don't do every patch every time, and for good reason. So, if in the course of their update cycles they don't patch, because they don't know the flaw(s) are there, they are pretty much screwed. Another fun Microsoft patching policy.
          • yeah

            But Apple can get away with silent patching because they are not vulnerable to attack, so it's ok that you apply this practice to Microsoft only, as usual, since everyone knows that if you get a Mac, you're done. Shady accounting deals, the most proprietary software available and an army of PR people to attack every discovered flaw and claim it's not (while patching it anyway from a "routine" code inspection) is great practice for the end users. <br>
            Spin it how you want, Vista is decidedly the safest OS in history. And it provides the richest feature set and programming environment, as it has since the first day the first version of Windows was released.
          • HAHAHA, yeah, just spin yourself right around

            and realize you are being lied to, plain and simple, If you are referring to Windows being secure by patch count you are seriously deluded.

            Have fun in your delusional fortress of delusional claims and ridiculous statements. Everything you say is a load. Apple with the most proprietary software available? Wowo you are deluded, that's all MS software is...oh and do you have links to back up anything you say?
            Kid Icarus-21097050858087920245213802267493
          • since you brought it up,

            you first. provide links that show I'm delusional about anything I said. <br>
            You do realize OS X can only run on a Mac and the source is proprietary, right? It doesn't matter if they are borrowing the BSD engine, they are a proprietary software company. Seen the source for iLife or any other Apple package lately? <br>
            At least with Windows you are licensed to run it on the hardware of your choice. Got a low budget and low computer needs? Vista basic on a 599.00 laptop or 499.00 desktop is just for you. Hardcore gamer and media enthusiast? Then Dell has just the machine for you. It costs more than a Mac but Dell has some models with significant more horsepower, graphics resolution and features than a Mac. You have the choice. You need a PC somewhere in the middle? Some stuff you want, some you don't? Well with Windows, Steve Jobs is not picking out your hardware for you, you can pick and choose exactly what you WANT your machine to have at the Dell or HP or Toshiba or Gateway or Lenovo (etc etc) site and get the machine you need, not the one Steve Jobs designed for you.
            Silverlight's engine, most standards compatible in the industry. IE's engine is getting close. Version 8 will be fully compliant. <br>
            And just think, Windows and it's features have been controlled by the DoJ and other large companies, like Google, who run crying to the DoJ and make Microsoft's search facility on Vista only be a choice in a list (probably the last choice) and Microsoft is forced to allow users to load Google desktop which is one gaping security hole with constant connection open to the internet you would not otherwise have 24/7 and some of your data gets to reside on Google's servers, whether you wanted it to or not, if you use their indexing feature (that used to be the only time, but probably all the time now). So Google has your data for as long as they want to retain it...your personal information, unless you don't keep any on your computer. Nice deal eh? People complain about microsoft innovation....Microsoft has had many great things shot down simply because they have the largest marketshare. Monopoly shonopoly, look it up, you have to have 100% control to be a monopoly and the liberal DoJ of that time, in their anti-capitalistic ways found they could just create new monopoly law and new area of computing that excluded Apple and thus Microsoft was guilty of monopolizing desktops that used a particular architecture. Judge Jackson said no other viable OS existed. OUCH, I bet Apple was happy to hear they didn't have a viable OS, and still don't based on the continuation of the ridiculous anti trust politics. <br>
            Anyway, point to where an objective 3rd party has proven that Vista is not the most secure OS in it's first 6 months compared to any other OS in their respective first 6 months of release. <br>
            And also point out how OS X is less proprietary than Windows please. <br>
          • Uhmmm OK?

            First off, let's see this silent patching accusation. You made the accusation, you prove it.

            Second your Windows "Security" means nothing because these flaw counts that MS has been dishing out are not the complete count. I'm not sure what other way you are assuming your "security", oh how about known exploits, like say the ani cursor exploit or the bot net problem, or how about, you get the idea.

            Third, somehow Windows is some open book? Dude get a clue, all of MS's software is wrapped in proprietary BS. So you can build your own computer and put MS on it, and that is somehow more open. Wow, that is just SO open. Vista is ripe with DRM goodness to the core, pal.

            But then again, it doesn't matter, you feel you need to unnecessarily rip into Apple for some reason. You and NonZealot make a great team. If you don't use their stuff why do you even CARE?!? I don't get it. I at least use Windows products so I have a leg to stand on when it comes to an objective opinion but you and Zelaot on the other hand speak from no experience whatsoever except this idiotic hatred towards them and a willful tendency to slander them outright, day in and day out. WHY?!?!

            Kid Icarus-21097050858087920245213802267493
          • Spin?

            It's you saying that Vista is the safest OS in history. That alone shows you are delusional.
          • well

            it's easy to be insulting and say I'm delusional, as is expected of you, but it's quite another thing for you to prove that it's not the safest OS. I'll be waiting for your detailed comparisons and conclusive evidence otherwise.
          • Yes, delusional

            1) They didn't count "silent flaws/fixes" only the disclosed ones
            2) They measured Vista against entire distributions
            3) You're incapable of reading, or unwilling to read through and discovering this yourself

            Conclusion. You're delusional.
          • wrong

            1. There is no proof of other silent patching. You can't assume Microsoft is the only player doing so. your point is moot, even for linux since many distros have proprietary code and packages. <br>
            2. The distribution is the distribution. They compared against what comes on the disc. In the case of Red Hat they did the counting over again against the standard workstation and not enterprise workstation. So you are wrong. Besides, XP and Vista come with IIS, MSMQ, COM+, MSDTC etc. <br>
            3. I don't care, nor could I ever, care what a tiny troll like you thinks about me. Your insults and childlike replies don't register. They are amusing at times, but mostly a sign of your ignorance.
          • Keep on spinning

            Maybe you will catch fire.

            You excuse the non-disclosed thing Microsoft does in these statistics. You cannot tell the difference between an OS and a complete distribution.
          • keep being trite.

            You are right, I don't know the difference between the kernel or subsystems or production packages. <br>
            So you define OS for me. I want to know if what comes on the Vista disc constitutes an OS only, and why. <br>
            Also, thanks for pointing out so forcefully that it's not just the Linux OS that is so buggy, but also all of the packages written to run on it.
  • He is full of it!

    [B]"When I asked Jones and other Microsoft security staffers about ignoring silent fixes in these reports ? which could significantly increase the Windows flaw count ? [U][I]they argued that everyone (including Linux distributions)[/U][/I] issues patches with silent fixes."[/B]

    Linux vendors don't have to hide fixes, they can't! The source code is open so someone would catch it and raise holy hell over it. Nice try, "Well we do it so everyone else must too!", sorry not everyone is a lowlife no morals and no sense of honor corporation. ]:)
    Linux User 147560
    • Pitifull attempt to distort the truth!

      If the fact that the source is open and you can't hide the flaws were true how did the flaws occur in the first place. The truth is it takes a trained eye to find flaws. They can hide right out in the open and never be found until someone looking to exploit it or to prevent an exploit finds it. Do I think that the people coding Linux discover issues and fix them without making them known? Absolutely! They would be stupid not to.