Can Mozilla's security metrics project end the patch-counting nonsense?
The idea is to develop a metrics model that goes beyond simple bug counts to accurately reflect the effectiveness of secure development efforts and the relative risk to users over time.
This is a real sore subject with me, especially because Microsoft uses patch counts to preach the gospel of its SDL (security development lifecycle), totally ignoring silent fixes and those security bugs that never gets patched until a "future service pack."
[ SEE: Skeletons in Microsoft’s Patch Day closet ]
With the meticulous Mogull on board to manage this new Mozilla project, I'm hopeful that a metrics model will emerge to help guide the entire industry.
Mozilla security chief Window Snyder explains:
Our goal in this first phase of the project is to build a baseline model we can evolve over time as we learn what works, and what does not. We do not think any model can define an absolute level of security, so we decided to take the approach of tracking metrics over time so we can track relative improvements (or declines), and identify any problem spots. This information will support the development of Mozilla projects including future versions of Firefox.
Mogull has released a spreadsheet (.xls) with a preliminary version of the model and Snyder is actively seeking feedback to make the project open and meaningful.
The final version will be a far more descriptive document, but for now we are using a spreadsheet to refine the approach. Feel free to download it, rip it apart, and post your comments. This is an open project and process. Eventually we will release this to the community at large with the hope that other organizations can adapt it to their own needs.
We would love to get your opinions on this, and if you are not comfortable commenting here you can mail Rich directly at rmogull@securosis.com. When we have reviewed the feedback, we will post here with findings and continue the effort with your help.
Once the project is complete, Snyder is hopeful that it will help to track security trends in the development of Firefox; measure the effectiveness of various tools, stages and techniques of secure development; and measure the exposure window when new vulnerabilities are discovered.
I'm just hoping that others are paying attention and we see an end to the silly patch-counting PR games.