Detailed Firefox and IE vulnerability report

When I wrote "Is the Firefox honeymoon over" two weeks ago, it stirred up a huge debate that continued for days.  Yesterday, Dan Farber interviewed the 20-year-old co-creator of Firefox Blake Ross in a podcast where he addressed the criticism of Firefox in the media and my blog in particular.  Ross criticized the analysis in my blog on two main points.

• No comparisons in the severity of the flaws
• No comparisons in the response time to fix the flaws.

On the first point, Ross is absolutely right that the severity of a vulnerability is critical.  Because I didn't set out to do a formal security analysis of Firefox and Internet Explorer, I created an overly simplified comparison.  Since this has sparked a huge debate, my original analysis is not sufficient to be conclusive.  Ross believes that Internet Explorer's vulnerabilities are much worse than Firefox vulnerabilities so we'll have to see if he's right.

On the second point, Ross believes Firefox is more responsive to security vulnerabilities and delivers more timely patches.  Ross even quoted that Internet Explorer had about "10 to 15" unpatched vulnerabilities, so we'll have to see if he's right.

To settle this debate, I created this detailed chart going back as far as the Secunia reports went for Firefox 1.x.  Because Ross had said to Dan Farber in an email that some patches were too low risk to be considered vulnerabilities (which I agree), I left out all vulnerabilities below Secunia's "moderate criticality" rating.  Then, I not only compared unpatched vulnerabilities, I also included statistics for all of Microsoft's unpatched vulnerabilities before August 2004.

I color coded the results. Grey highlighting signifies the security loser of the month and Red characters signify unpatched vulnerability.  All data was compiled from Secunia on 9/25/2005.

Now that we have the detailed comparison, let's see how Firefox and Internet Explorer fared.

On the issue of vulnerability severity, it appears that either camp can claim victory depending on when you look at the numbers.  If you look at vulnerability activity before March of 2005, Microsoft Internet Explorer had a consistent drip of monthly vulnerabilities and a huge rash of problems in October 2004.  During that same period of time, Firefox was fairly quiet.  After March of 2005, the trend reversed and Firefox had a continuous drip of monthly vulnerabilities while Internet Explorer was relatively quiet.  Internet Explorer appears to have had an ugly history but seems to be maturing and stabilizing while Firefox appears to be going through some growing pains in the last seven months.  From these results, it is clear is that there is no clear victor and neither camp has anything to be proud of with all these security vulnerabilities.

On the issue of patch responsiveness, Ross appears to have a point in that Firefox holds an edge in this department.  While it isn't the "10 to 15" unpatched Internet Explorer vulnerabilities that Ross talked about if you exclude the low risk flaws that were omitted for both browsers in this comparison, Microsoft has five "moderately critical" issues that have not been addressed yet.  There is even a "highly critical" vulnerability from October 2003 that Microsoft has not addressed yet, but I'm not sure if this old vulnerability still applies to Windows XP SP2.  Some of the "moderately critical" vulnerabilities had example exploits that you can test and they failed to produce any results on my Windows XP SP2 system.  I'll try to get an answer from Microsoft about all these unpatched vulnerabilities and post their response in a follow up.

The bottom line is that we have some mixed results where either browser camp can claim victory.  I will wrap this blog up by saying that Blake Ross has produced some important innovations in his web browser and I want to thank him for his hard work and I wish him luck in his new endeavors.  Microsoft was getting overly complacent with Internet Explorer and it took a jolt from Firefox to light a fire under their feet.  Even the staunchest supports of Microsoft will have to admit that they are benefiting from renewed competition in the web browser market.  I also congratulate Microsoft for cleaning up Internet Explorer and I hope their efforts to clean up their code continues in all Microsoft products.