How clean is open source? DHS-funded study has some answers
The first results of a study looking at the quality of open source code running on government computers found an average of half a bug per 1,000 lines of code, says Coverty Inc. The study, funded by the Dept. of Homeland Security and conducted by Coverity Inc., Stanford University and Symantec Corp., found even fewer bugs in core applications like Linux and Apache. Here are some results for the most important applications:
- Apache, .250/1000
- Firefox, .348/1000
- Gnome .461/1000
- Linux-2.6 .322/1000
- Perl .091/1000
- PHP .475/1000
- Samba .665/1000
Comparisons can't really be made with proprietary software, though, Coverty CTO Ben Chelt says. According to Washington Technology:
Generally speaking, it is difficult to determine how well these open-source programs compare with their proprietary counterparts, Chelf said. Coverity has tested only a few commercial products, so direct comparisons cannot be made.
The company has drawn a number of observations from the study, and elaborated upon them in a paper accompanying the results (available on the Coverity site after registration). The chief lesson is that the number of lines of code is not an indicator of quality. Smaller programs can have plenty of bugs while larger projects, such as the Linux kernel, can be tightly controlled. Quality is more accurately reflected by the ratio of developers to the size of the code base and by the number of users who use the software (and provide feedback).
