The irony isn't lost on me: Coverity, a a company specializing in software quality and security testing solution, has found that open source software has fewer defects in its code than proprietary programs in the aftermath of open-source OpenSSL Heartbleed programming fiasco. Nevertheless, the numbers don't lie and the 2013 Coverity Scan Open Source Report (PDF Link) found that open source had fewer errors per thousand lines of code (KLoC) than proprietary software.
The Coverity Scan service, which the study was based on, was started with the US Department of Homeland Security in 2006. The project was designed to give hard answers to questions about open source software quality and security.
For this latest Coverity Scan Report, the company analyzed code from more than 750 open source C/C++ projects as well as an anonymous sample of enterprise projects. In addition, the report highlights analysis results from several popular, open source Java projects that have joined the Scan service since March 2013. Specifically, the company scanned the code of C/C++ programs, such as NetBSD, FreeBSD, LibreOffice, and Linux, and Java projects such as Apache Hadoop, HBase, and Cassandra.
The 2013 report's key findings included:
- Open source code quality surpasses proprietary code quality in C/C++ projects. Defect density (defects per 1,000 lines of software code) is a commonly used measurement for software quality, and a defect density of 1.0 is considered the accepted industry standard for good quality software. Coverity’s analysis found an average defect density of .59 for open source C/C++ projects that leverage the Scan service, compared to an average defect density of .72 for proprietary C/C++ code developed for enterprise projects. In 2013, code quality of open-source projects using the Scan service surpassed that of proprietary projects at all code base sizes, which further highlights the open source community’s strong commitment to development testing.
- Linux continues to be a benchmark for open source quality. By leveraging the Scan service, Linux has reduced the average time to fix a newly detected defect from 122 days to just six. Since the original Coverity Scan Report in 2008, scanned versions of Linux have consistently achieved a defect density of less than 1.0. In 2013, Coverity scanned more than 8.5 million lines of Linux code and found a defect density of .61.
- C/C++ developers fixed more high-impact defects than Java developers. The Coverity analysis found that developers contributing to open source Java projects are not fixing as many high-impact defects as developers contributing to open source C/C++ projects. Java project developers participating in the Scan service only fixed 13 percent of the identified resource leaks, whereas participating C/C++ developers fixed 46 percent. This could be caused in part by a false sense of security within the Java programming community, due to protections built into the language, such as garbage collection. However, garbage collection can be unpredictable and cannot address system resources so these projects are at risk.
- Apache HBase serves as benchmark for Java projects. Coverity analyzed more than eight million lines of code from 100 open source Java projects, including popular big data projects Apache Hadoop 2.3 (320,000 lines of code), HBase (487,000 lines of code), and Apache Cassandra (345,000 lines of code). Since joining the Scan service in August 2013, Apache HBase — which is Hadoop’s database — fixed more than 220 defects, including a much higher percentage of resource leaks compared to other Java projects in the Scan service (i.e., 66 percent for HBase compared to 13 percent on average for other projects).
Zack Samocha, senior director of products for Coverity, said in a statement, "Our objective with the Coverity Scan service is to help the open source community create high-quality software. Based on the results of this report — as well as the increasing popularity of the service — open source software projects that leverage development testing continue to increase the quality of their software, such that they have raised the bar for the entire industry."
Coverity also announced that it has opened up access to the Coverity Scan service, allowing anyone interested in open source software to view the progress of participating projects. Individuals can now become Project Observers, which enables them to track the state of relevant open source projects in the Scan service and view high-level data including the count of outstanding defects, fixed defects, and defect density.
"We’ve seen an exponential increase in the number of people who have asked to join the Coverity Scan service, simply to monitor the defects being found and fixed. In many cases, these people work for large enterprise organizations that utilize open source software within their commercial projects," concluded Samocha. "By opening up the Scan service to these individuals, we are now enabling a new level of visibility into the code quality of the open-source projects, which they are including in their software supply chain."