Mozilla is working on an effort to figure out a new way of measuring how secure Firefox actually is over a period of time. While I think on the surface that it’s a good idea I also have serious doubts about the value that the actual metric will have in the end. After all Cisco, Oracle and others have already kinda/sorta figured this out in an industry standard way already.
Window Snyder, Mozilla’s head of security wrote in a blog post:
We are trying to develop a model that goes beyond
simple bug counts and more accurately reflects both the effectiveness
of secure development efforts, and the relative risk to users over time.
OK then let’s step back a second. What’s wrong with simple bug counts for one?
For any given Mozilla security advisory there are one or more CVE identified issues. Each one of those CVE identified issues could have one or more bugzilla entries attached to them. So doing the simple math here would imply that a single Mozilla security advisory could fix multiple bugs. The Mozilla advisories already do a fine job of grouping multiple related CVEs into one issue.
Does the fact that a vendor – any vendor – fixes more or less bugs or issues or issues more or less security advisories make them any more or less secure?
Mozilla fixes a lot of bugs in each release, that doesn’t mean that the release was necessarily insecure to begin with. But simply counting bugs, CVEs and advisories is a simple yet incredibly realistic method for measuring the security related activity on a given project.
YES I understand that due to the fact the Mozilla fixes so many bugs that some lame reporter (or competitive vendor) could use the numbers to imply something negative. Those same numbers can be used to show progress in a positive light as well.
Certainly understanding relative security is a great idea as well as understanding the true impact and severity of bugs. That’s why two of the biggest technology vendors on Earth, Oracle and Cisco use the CVSS system (The Common Vulnerability Scoring System) which defines the severity of vulnerabilities.
If I ran security for Mozilla I’d look very seriously at CVSS as the basis for a security risk matrix and as a core metric to help gauge the relative security of Mozilla products over time. It’s an industry standard approach that wouldn’t require Mozilla to re-invent the wheel.
They may still have to build out the whole car mind you, but at least the core vulnerability metrics would enable an apples to apples (no pun intended) comparison with other vendors.