Flaw counting comparisons useful but fall short of true picture

The Windows vs Linux security report card that I wrote about from TechEd two weeks ago is officially out and Microsoft has stepped up its PR campaign to argue that Windows Vista has a "lower vulnerability fix and disclosure rate" than competitive Linux distributions.

Jones released the study (download PDF) and posted a primer with details on the methodology used to compare vulnerabilities disclosed and fixed in the first six-month period after a product ships.

In all four cases studied for the 6 month period after ship, Windows Vista appears to have a lower vulnerability fix and disclosure rate than the other products analyzed, including the reduced Linux installations. This affirms the early results that we found after 90 days and provides a supporting indicator that the Microsoft Security Development Lifecycle process and heightened focus on security is having a positive impact on Microsoft Windows in terms of fewer vulnerabilities.

He also, for the first time, broke out "high severity" vulnerabilities in the comparison and again Jones found that Windows Vista and even Windows XP fared better than Linux distribution workstations.

The controversial studies have been dismissed as biased propaganda -- see Talkback comments here and here -- from Redmond (Jones is security strategy director in Microsoft’s Trustworthy Computing group) but in my mind it's a useful attempt to dig into the publicly available numbers to find a measurement.

The problem I have with Jones is that his flaw counting ignores silently fixed vulnerabilities and makes assumptions on security based only on publicly documented vulnerabilities.

As a policy, Microsoft routinely ships silent fixes within its security bulletins if flaws are discovered internally. These are never assigned CVE numbers and will never appear in these comparison reports from Jones.

When I asked Jones and other Microsoft security staffers about ignoring silent fixes in these reports -- which could significantly increase the Windows flaw count -- they argued that everyone (including Linux distributions) issues patches with silent fixes. Additionally, Jones claimed that vulnerabilities discovered and fixed without help from external researchers do not put anyone at risk since they are only found internally.

This argument ignores the dramatic rise in zero-day attacks that use undocumented flaws/exploits to target .gov, .mil and other business networks. Try telling an enterprise that's been hit with a zero-day that his loss is less important because it's not a widespread risk, you just might get a punch in the nose.

So, while Jones' reports make for good discussion fodder, take them with a grain of salt. Hey, even Jones admits that he's biased.