A new study by code analysis firm Klocwork has discovered new flaws in open source programs that previous scans by a Department of Homeland Security-sponsored study did not.
Apparently, the open source projects in question were notified by Klocwork of their results, but at least one open source vendor disputes the claim.
Klocwork, which this week released its Klocwork K7.1 automated defect and vulnerability source code scanner, ran its application against the Amanda 2.5, Samba 3.0.23, and XMMS 1.2.10 open source projects.
The scan apparently found hundreds of defects and vulnerabilities in the three projects they analyzed.
“Interestingly, our analysis was a follow-on to the use of another static analysis tool,” Klocwork Product Marketing Manager Brendan Harrison wrote in an unpublished blog posting sent to internetnews.com.
“The first analysis had been undertaken as part of a U.S. Department of Homeland Security initiative to analyze open source code. Despite the internal testing as well as the first static analysis that had identified numerous defects, we found there were still more bugs lurking in the code bases.”
Harrison explained to internetnews.com that the analysis was against code that Coverity reported as having zero defects.
Within that code, Klocwork found a variety of defect types that can be summarized in the following high-level categories: NULL Pointer Deference Issues, Usage of Freed Memory Issues, Array Bounds Violations (which can also be Buffer Overflows depending on the situation), Unvalidated Data Issues and Injection Flaws.
The “first” analysis that Klockwork is referring to is being completed by source code analysis firm Coverity under the auspices of a DHS grant.
Some of the initial results of the study showed that LAMP (Linux/Apache/MySQL, PHP/Perl/Python) code quality was higher than a baseline of 32 open source projects.
Previous non-DHS backed studies from that followed Coverity studies have reported low defect densities in the Linux kernel and in the MySQL database.
Ben Chelf, Coverity’s CTO, disputed Klocwork’s claims about the Coverity results.
“They mentioned that another vendor claimed that the code was defect-free and we think that is a misrepresentation,” he told internetnews.com.
“We never claimed that we find all the defects in a particular piece of software, our source code analysis finds an awful lot of defects and has a low false positive rate.”
Chelf admitted that he was unaware of the specifics of the Klocwork scan, its methodology or its interaction with the open source communities whose code was scanned.
A Klocwork spokesperson told internetnews.com that Klocwork submitted all of its findings to the open source projects to help them correct the errors. The spokesperson noted that the Samba project provided a quote for the release, and were very eager to fix the defects.
Zmanda, which is a corporate sponsor for the Amanda project and leads its technical development, said it was not contacted by Klocwork.
“Since the press release went public, we have made contact with Klocwork and they have agreed to share the test results with us,” said Ken Sims, vice president of business development and marketing for Zmanda.
The initial feedback from Klocwork, according to Sims, is that the Klocworks tool did find flaws in the Amanda code that were not found by Coverity.
“However, we are not in a position to validate or comment on these flaws until we have an opportunity to review the results,” Simms said.
Regardless of who was told what and when, the bottom line is all about achieving optimal code quality. Sims said once they have the chance to review the Klocwork results of valid bugs or code defects discovered in Amanda, Zmanda will make it a top priority to fix the bugs as soon as possible.
“We welcome independent analysis of Amanda; the scrutiny of open source code is one of the true benefits of the open source development model and ensures high quality software,” Sims said.
Coverity’s CTO also called the Klocwork effort a positive move.
“The open source community benefits by having as many vendors as possible going over its code,” Chelf said. “We encourage more and more people to put their results up on line and let the open source community use them the same way we did.”
Coverity’s DHS-sponsored effort is on the verge of hitting a major milestone that proves Chelf’s point about providing code analysis to open source project.
“We’re getting close to 5000 defects fixed as the result of defects we found in open source code,” Chelf said. “Just to see that many bugs being fixed is a testament to the open source community’s response to any data about their code. They care about their code and that’s obvious.”