Mozilla: Critical vulnerability in Microsoft flaw-counting
Snyder (left), a former Microsoft security strategist, said Jones use of publicly available data in his side-by-side comparison of the two browsers is not an accurate measurement of a browser's security profile.
"Unfortunately for Microsoft (and for anyone trying to use this report as analysis of useful metrics) he does not count all the security issues. If he were able to count them all, Microsoft could get credit for all the bugs they fixed. He counts only the public issues, because that is all Microsoft will tell us about. Microsoft is worried that if it ever says it has fixed X security issues, the world will focus on that it had X vulnerabilities in the first place, not that they are now fixed and no longer a risk for users," Snyder said in a hard-hitting response to Jones's study.
Snyder, a pen-testing specialist who was responsible for security sign-off for Microsoft's Windows XP SP2 and Windows Server 2003, argues that the data used by Jones is a "small subset of all the vulnerabilities" affecting Internet Explorer.
[ SEE: IE vs Firefox: Microsoft crunches security numbers ]
"[The] vulnerabilities that are found through the QA process and the vulnerabilities that are found by the security folks they engage as contractors to perform penetration testing are fixed in service packs and major updates. For Microsoft this makes sense because these fixes get the benefit of a full test pass which is much more robust for a service pack or major release than it is for a security update," she explained.
However, Snyder adds, this means IE users have to wait sometimes a year or more to get the benefit of the QA work.
"That’s a lot of time for an attacker to identify the same issue and exploit it to hurt users. Sometimes it just takes time to put in a complicated fix. Anyone that has shipped a major piece of software can relate to that. But this is not the case for every internally found security issue. Extending this process to include fixes that are ready and just sitting on the tree waiting for the preferred vehicle to ship increases risk for users. But it sure keeps those bug count numbers down," she added.
[ SEE: Firefox narrows patch deployment window ]
"If we as an industry would just acknowledge that counting bugs is useless then vendors could feel safe talking about what they are doing to protect users. At Mozilla we fix our bugs openly. When you count Mozilla security bugs you are seeing not just those that are reported externally, but also the ones that would be considered internal if we acted like most other software vendors," Snyder said.
Mozilla vice president of engineering Mike Schroepfer also used his blog to offer a sharp response to Jones and call attention to the absence of real data on actual bugs affecting Microsoft products:
[T]here is no way for anyone outside of Microsoft to confirm how many vulnerabilities ever existed in Internet Explorer. In an earlier post the author of the study touts the benefits of the Software Developement Lifecycle (SDL) at Microsoft as a reason Vista is more secure. Surely one of the goals of this process is to identity and fix security bugs right? How many bugs were identified and fixed using the SDL during development? Your guess is as good as mine.
"Bug counts are meaningless, what matters is whether you are at risk or not," Schroepfer declared.
Instead of counting bugs, Mozilla has long suggested that the time it takes to release -- and deploy -- software patches should carry more weight. Snyder has proposed a "time to deploy" metric a better way to measure a software vendor's approach to securing customers.
"Time to deploy" is the length of time it takes for users to get a patch installed once the fix is available from the vendor. This in effect gives Firefox a major advantage over IE because the browser's default auto-updating mechanism significantly cut down on the time it takes to push a security upgrade down to end users.