Can Mozilla's security metrics project end the patch-counting nonsense?

Can Mozilla's security metrics project end the patch-counting nonsense?

Summary: In partnership with indie security consultant Rich Mogull (left) Mozilla has launched a valuable Security Metrics Project that could help to -- we can only hope -- put an end to the silly notion that patch-counting helps to determine a product's security posture.The idea is to develop a metrics model that goes beyond simple bug counts to accurately reflect the effectiveness of secure development efforts and the relative risk to users over time.

SHARE:
TOPICS: Security, Browser
10

Can MozillaÂ’s security metrics project end the patch-counting nonsense?In partnership with indie security consultant Rich Mogull (left) Mozilla has launched a valuable Security Metrics Project that could help to -- we can only hope -- put an end to the silly notion that patch-counting helps to determine a product's security posture.

The idea is to develop a metrics model that goes beyond simple bug counts to accurately reflect the effectiveness of secure development efforts and the relative risk to users over time.

This is a real sore subject with me, especially because Microsoft uses patch counts to preach the gospel of its SDL (security development lifecycle), totally ignoring silent fixes and those security bugs that never gets patched until a "future service pack."

[ SEE: Skeletons in Microsoft’s Patch Day closet ]

With the meticulous Mogull on board to manage this new Mozilla project, I'm hopeful that a metrics model will emerge to help guide the entire industry.

Mozilla security chief Window Snyder explains:

Our goal in this first phase of the project is to build a baseline model we can evolve over time as we learn what works, and what does not. We do not think any model can define an absolute level of security, so we decided to take the approach of tracking metrics over time so we can track relative improvements (or declines), and identify any problem spots.  This information will support the development of Mozilla projects including future versions of Firefox.

Mogull has released a spreadsheet (.xls) with a preliminary version of the model and Snyder is actively seeking feedback to make the project open and meaningful.

The final version will be a far more descriptive document, but for now we are using a spreadsheet to refine the approach. Feel free to download it, rip it apart, and post your comments. This is an open project and process.  Eventually we will release this to the community at large with the hope that other organizations can adapt it to their own needs.

We would love to get your opinions on this, and if you are not comfortable commenting here you can mail Rich directly at rmogull@securosis.com.  When we have reviewed the feedback, we will post here with findings and continue the effort with your help.

Once the project is complete, Snyder is hopeful that it will help to track security trends in the development of Firefox; measure the effectiveness of various tools, stages and techniques of secure development; and measure the exposure window when new vulnerabilities are discovered.

I'm just hoping that others are paying attention and we see an end to the silly patch-counting PR games.

Topics: Security, Browser

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

10 comments
Log in or register to join the discussion
  • Can Mozilla's security metrics project end the patch-counting nonsense?

    I only had a quick look at the spreadsheet so I may be wrong. However it looks like this is bug counting exercise with a bit of 'how long were we exposed?' thrown in. Maybe it's a bit better than patch counting but it doesn't seem significantly better.
    SamYeager
    • Still, if we count patches, then companies will be encouraged to NOT

      release patches for know problems, and keep them quiet. For instance, pretending they do not exist and rolling them into service packs, is NOT in the customers interest, but helps the vendor pretend his product is more secure. A high patch count might actually be a very good thing.
      DonnieBoy
  • RE: Can Mozilla's security metrics project end the patch-counting nonsense?

    Well, putting exposure time as a factor shows they recognize that it's important to fix these things in a timely manner. It's one thing to leave your house unlocked to walk to the corner shop, it's another to go on vacation without locking it.
    Hrothgar - PCLinuxOS User
    • Bleak House

      Perhaps I should speak after reviewing the proposed
      formula, but I do have these initial comments. It feels to
      me that trying to formally inspect code and quantify its
      security/functional correctness is a restatement of the
      Turing Halting Problem.

      To (after a fair, independent, and rational assessment)
      declare one program more secure than another is to say
      what, exactly? Because vulnerabilities and exploits are a
      weakest link type of problem, a program with 1 easily-
      exploited vulnerability may be far more risky to use than
      another code with 20 vulnerabilities.

      What I say is well understood and the various metrics used
      by responsible parties attempt to mitigate the issue.
      Vendors, though, want to interpret the data so as to gain a
      marketing advantage. So, here, then, is the next major
      problem. Patch counts may be gamed by obscuring the
      how many fixes are covered by a patch release. There's an
      incentive to lower the risk assessment and to withhold
      disclosure of under-repair issues so no one starts a clock
      on their patching speed. Security firms who don't play ball
      are cut off from access, to the degree possible.

      It's a problem, and I don't see where the vendors would
      have any interest in changing a system which allows them
      and their pr minions to bash the competitor. Any
      consumers' union type of organization will be EULAed and
      NDAed into ineffectiveness.
      DannyO_0x98
  • In the end, it's money and resources

    So I'll be taking my browser from the people who wrote the global OS, have the best programmers in the world and spend a lot of time on testing and quality control - also having the bulk of the world using the products helps find any other bugs/exploits that slip through.

    Apart from the no-brainer of which browser to use (hint: it comes with Windows and it's free), I'll take money and resources every time over passion and idealism. So forget the patch count (we have to anyway, since Mozilla has far too many again) ;-)
    tonymcs@...
    • So, where's your Microsoft rep and MCSE flunkies?

      You really are doing a poor imitation of the proverbial "Mike Cox."
      zkiwi
    • Illogical rant .....

      #1- If MS actually hired "the best programmers in the world" their software would actually be a hell of a lot better. The fact is MS hires the CHEAPEST programmers in the world, not the best.

      #2- Mozilla has very few vulnerabilities, it is just that idiots like to add normal (non-security) bugs and RFEs to the count since they have 100% access to the Bug reports, but no clue on what they read.

      #3- I will take money too .... and that is what MS is stealing from me by having so many highly critical vulnerabilities unpatched for years. Or have you forgotten about the 10-20 vulnerabilities that are still unpatched since 1998???
      wackoae
  • Just Another Gaming System ... So That Mozilla Appears to Win

    Mozilla is well knowns for releasing patches with little or no testing (or planning), that have to be re-patched in days. They are also well known for breaking a sizeable percentage of their add-ins with very patch.

    To me, that says they are operating totally out of control. but by their proposed metrics, they WIN - by knee-jerking out "patches" to reduce the window of exposure.

    EVERYONE with an understanding of security vulnerability knows this basic fact -- reducing the amount of redundant code on a machine reduces the security risks. Until such a time that Internet Explorer can actually be removed from a Windows system without breaking the help system, the control Panel, Windows Update, donloading media, etc., then ADDING FIREFOX to Windows increases the security problems by adding a redundant web browser to the attack surface.

    By the logic above, the same argument can be made for Linux (unless FireFox is the distribution default) and Apple computers.
    PMC-CON
    • Attack surface.

      PMC-CON wrote:

      [i]EVERYONE with an understanding of security vulnerability knows this basic fact -- reducing the amount of redundant code on a machine reduces the security risks. Until such a time that Internet Explorer can actually be removed from a Windows system without breaking the help system, the control Panel, Windows Update, donloading media, etc., then ADDING FIREFOX to Windows increases the security problems by adding a redundant web browser to the attack surface.[/i]

      Good point. Rhetorical question: why exactly was IE so tightly bound to the OS?

      [i]By the logic above, the same argument can be made for Linux (unless FireFox is the distribution default) and Apple computers.[/i]

      Not quite. Linux help systems, update management, etc. are not bound to a particular browser or rendering engine. KDE uses Konqueror for much of this, gnome uses gecko (but not the same code Firefox's gecko uses), and it is entirely possible to use and manage linux without any GUI at all. Help (man pages, info pages), package management and updates can all be done from the command line.
      JDThompson
      • Rhetorical Question Answered

        I will offer that Windows was originally designed for consumer-oriented resource-constrained PCs in a time when trusted computing meant that only your floppy drive offered any risk for infection. (OS/2 was supposed to be the business version, then NT.)

        Sharing of components to perform different functions within Microsoft ostensibly reduced development time, lowered development costs and reduced the price and increased the performance of a given set of hardware (RAM and hard drive costs) running a given set of services. Adding Internet Explorer to the mix was done in the same model, even though it opened Pandora's box, in a way that is the fairy tale foretold.

        Why this model persists today in partially due to backward compatability and partly due to inertia on the part of Microsoft. (I think they keep focusing on the jewel of Hope at the bottom of the box. BTW, it's made them rich beyond their dreams; go figure?) However wise, they have patched and patched, rather than starting from scratch. IE7 protected mode, mandatory automatic updates, PatchGuard and UAC are some of the current solutions for protecting the architecture.

        Konqueror and gecko are in some ways similar to the componentization I see from IE, in the weaker way I understand it. Konquerer is web browser and file browser at least. If Konqueror is conquered I bet a (KDE) Linux box goes down in flames pretty fast.

        I did not mean to imply that Firefox was embedded in Linux; I meant to say that that adding Firefox to a Linux distribution where it is not the default browsr increases the attack surface for that box, just as adding it to Windows does. If the distro has made it the default browser, then presumably they will support it better that say Kubuntu does/did when I added Firefox (clumsily) to a PC I was testing; that is to say, I would be able to use the normal update tools to get a secure version from the distro source, rather than having to add a secure version out-of-band from the mozilla site.

        Lots of Windows PC feature are configuable or usable from the command line, but it is simply awful to use it (as are the cretinous man pages in Unix for instance.) That's why we DOS-folks switched to Windows instead of Unix, to get away from the impossibly inefficient CLI.
        PMC-CON