Measuring Computer Security
Metricon 1.0 took place in Vancouver earlier this month. I wasn't able to be there in person, but I've been reviewing the material that the conference produced. The premise of the conference is pretty simple: we need better metrics for computer security.
Andrew Jaquith of the Yankee Group and Steven Bellovin of Columbia University delivered keynotes that took a point-counterpoint approach on metrics.
You can access all the slides from the presentations individually. It can often be difficult to make sense of a talk purely from the slides, but I found Tim Geer's presentation to a USENIX session the day before Metricon to be very readable and informative. Tim includes notes on many of the slides and that helps. Be warned, it's very long (346 slides).
In setting up the whole discussion, Tim mentions an interchange with a CISO of a major Wall Street bank who said "Are you security people so stupid that you cannot tell me...."
- How secure am I?
- Am I better off than this time last year?
- Am I spending the right amount of $$?
- How do I compare to my peers?
- What risk transfer options do I have?
Every other part of the bank could answer similar questions to five digit accuracy, so why couldn't security professionals?
An article in CSO Online goes over five measurements from Andrew Jaquith that you could use right now in your business (see the article for details):
- Baseline Defenses Coverage (Antivirus, Antispyware, Firewall, and so on)
- Patch Latency
- Password Strength
- Platform Compliance Scores
- Legitimate E-Mail Traffic Analysis
Geer mentions these and others in his presentation. These are informative without being overly complex. That said, Jaquith isn't fond of overly simplistic models based on red, yellow, and green dots as substitutes for real data.
I've found over and over again that people pay attention to what you measure. That's good and bad. If you measure the wrong thing, people will pay attention tot he wrong thing. But by carefully picking metrics and putting in place a methodology for generating and reviewing them, you can go a long way toward helping an organization pay attention to what matters.