Mozilla and Microsoft and how we measure security flaws
I had the opportunity this week to speak with Window Snyder who just joined Mozilla as the lead security strategist. Snyder was most recently a principal founder and CTO at Matasano Security and was a senior security strategist for Microsoft heavily involved in their security development lifecycle. It's no wonder why Mozilla would want to downplay the number and frequency of exploits. To kick off her new role at Mozilla, Snyder gave me a presentation that sought out to change the way people measure security.
The key issue raised by Snyder was that the current metrics for evaluating the security of a product is flawed.
- The current metrics that the industry uses to measure the security of a product is based on the number and frequency of vulnerabilities in a product
- Commercial vendors don't always patch everything
- Commercial vendors patch flaws through service packs and version upgrades which may hide the actual number of flaws
Citing people like Dan Geer and Allen Jones, Snyder and Mozilla believes that security metrics should be based on the following factors.
- Days of risk (time between disclosure and patch)
- Transparency of the patch process
- Security of the architecture
- Scope of fixes
I raised some issues and objections during the presentation because I don't believe that the "current method" based on how Mozilla defines them are completely flawed. For example, how can we not look at the frequency and number of critical remotely exploitable flaws in a particular product? A product that has a constant and plentiful stream of remotely exploitable flaws regardless of how good the "transparency" or the "architecture" is still stinks no matter how much perfume you spray on it. While Microsoft has had a steady stream of problems with Internet Explorer, there is no getting around the fact that Mozilla Firefox within the last year has had a higher number and frequency of exploits. It's no wonder why Mozilla would want to downplay the number and frequency of exploits in a particular product.
I should point out that Opera has the fewest number of exploits compared to Internet Explorer or Firefox by far.
As for service packs and new versions hiding the true number of exploits in a product, I pointed out that we haven't seen a service pack or new version of Internet Explorer for the last two years so this argument simply doesn't hold any water. Snyder pointed out that Vista and IE7 was coming out but failed to explain how that would affect the vulnerability metrics on IE6 for the last two years since IE7 and Vista didn't exist during that time. If a vendor fixes a product through internal testing before it's ever shipped to the customers, that's a good thing. As far as all the vulnerabilities counted against Firefox is concerned, they were all counted on shipped products.
As for the "days of risk" metric, that is a valid but it shouldn't be solely be based on the time between disclosure and patch. The actual "days of risk" should include some fraction of the life time of the product because the vulnerability doesn't just start when the vendor finds out about it. There is an entire black market for undisclosed vulnerabilities. The number and frequency of exploits along with the total number of days of risk both contribute to the vulnerability surface area.
Mozilla doesn't officially follow a monthly patch cycle like Microsoft but the reality is that they're pretty close to one if you look at their actual pattern of patching within the last year. Mozilla on average patches a little faster than Microsoft for critical vulnerabilities while Microsoft tends to stick to a strict monthly patch cycle even when there is an imminent threat looming with active exploits in the wild. Microsoft will occasionally issue an out-of-cycle patch if there is enough pressure on them in the media such as the case of the WMF exploit. But when there isn't that much pressure on Microsoft, they're content to wait for the next monthly update. I personally have always agreed on the need for an orderly monthly patch cycle because there is no way corporations can deal with daily or weekly patches, but any remotely exploitable vulnerability should be patched immediately if the exploit methodology is already publicly known.
Snyder's presentation included quotations from Brian Krebs who is a Washington post blogger that Microsoft takes 135 days to address problems while Mozilla takes 21 days though I don't really know how Krebs derives the 135 number. If we look at the recent zero-day Microsoft Office exploits that have been released the day after patch Tuesday, Microsoft has been issuing patches on the following patch Tuesday which is only 30 days away. That's still much too long to wait when exploits are in the wild as far as I'm concerned but it's certainly no where close to the 135 day figure. There have been some less critical issues that have been left unpatched (I pointed out a whole list of them here from Secunia) and some of them have come back to haunt Microsoft when some of those less critical vulnerabilities became very critical. Microsoft pointed out that a lot of those "unpatched vulnerabilities" were actually patched via service packs or weren't really vulnerabilities and I've verified some of that to be true, but Microsoft has failed to issue a detailed response to each of those alleged unpatched issues and it leaves them open to criticism.
The reality is that Microsoft's reputation for being lazy when it comes to patching software is sometimes deserved though it's not as bad as the 135 day figure. The fact that Microsoft was able to issue a patch for the Windows Media DRM crack FairUse4WM in a record 3 days doesn't help Microsoft's image either. The bottom line is that neither Microsoft nor Mozilla has something to be proud of to date when it comes to their security track record. The good news is that they're both competing so fiercely that they're each working harder to produce a more usable and secure product.