Nearly two and a half years ago, the US Department of Homeland Security
(DHS) issued a multi-year grant to help improve open source code quality. It appears
that the DHS investment has paid off.
According to a report from code analysis vendor Coverity, the DHS sponsored
effort has helped to reduce the defect density in 250 open source projects
by 16 percent over the past two years. That defect reduction translates into
the elimination of over 8,500 defects. The report on the benefits of the DHS
open source security efforts comes at a time when open source software is
increasingly becoming part of critical infrastructure both in the government
and in US enterprises.
“The improvement of project defect density is such that when we started the
effort they were at 0.30 defects per thousand lines of code and now they are
down to on average 0.25 defects per thousand lines of code,” David Maxwell,
open source strategist for Coverity told InternetNews.com. “I know
that feels like a small percentage change but when it’s over 55 million code
it adds up.”
Coverity is a code analysis vendor and runs its scanning tools on the
included open source projects to identify coding errors.
While many projects have benefited from running the DHS sponsored Coverity
scan, not all have actually managed to reduce their defects.
“There is a graph in the report that shows some project have significant
improvements and some that haven’t been actively using the results from the
scan that have actually increased in defect density,” Maxwell commented.
The report graph that was provided to InternetNews.com
doesn’t fully name the names of those project that did not improve as a
result their defect densities. The report however did identify Perl, PHP,
Python, Postfix, Samba and TCL among the projects that have been able to
reduce their code defect densities by using data from the Coverity scans.
Popular errors
Coverity’s scanning efforts have also provided them with some interesting
statistical data points about which type of errors seem to occur more often
than others. Leading the pack are Null Pointer Dereferences at nearly 28
percent.
“This type of error often occurs when one code path initializes a pointer
before its use, but another code path bypasses the initialization process,”
Coverity said in its report. “Because pointers are often used to pass
data structures by reference between pieces of program logic, they may be
the most commonly manipulated data objects due to repeated copying, aliasing
and accessing. Therefore, it is not surprising that the most frequently used
artifacts will incur the most errors in manipulation.”
Coming in second for recurring defects are resource leaks at 26 percent of
all defects.
“Resource leaks often involve failure to release resources when the initial
allocation succeeds, but a subsequent additional required resource is not
available,” the Coverity report explains.
Maxwell commented that in looking at the volume of data they were able to
collect on open source projects there were some things that went against
their expectations.
Among the findings that Coverity did not expect is one that has to do with
function length and its relationship to defect density. Maxwell explained
that there is a common myth or misconception that if a code function is too
long and for example can’t fit on a programmers screen, then it is more
defect prone. As it turns out, that’s not the case.
“The defect density did not increase,” Maxwell said. “There is almost no
correlation between average function length in a project and a project’s
defect density.”
Page 2 of 2
The DHS grant money that is helping Coverity to run the scans was originally
announced to only be for a three year term. The term ends in the
July/August time frame of this year.
“DHS does not have a mandate for doing ongoing efforts as they’re oriented
on research and development and technology transition so it wouldn’t be
within their purview to fund an ongoing effort,” Maxwell said.
That said, Coverity isn’t necessarily done with the open source scanning
effort.
“It is our intention to continue,” Maxwell added. “Open Source developers
have given us a very positive response and we want to continue to offer the
service.”
