Code quality in a version of the MySQL open-source database was found to be six times superior to that of comparable proprietary code, according to a recent study of
open-source software products by tech development firm Reasoning.
The results of the study come at a time when fierce debate rages as to
whether open-source software, such as the Linux operating system, is safer, or more secure than proprietary products such as Microsoft Windows.
Although Windows has been the favorite target of criticism over its numerous security patches in recent years, commercial software proponents and even analysts note that open-source software can also be less secure than its commercial counterparts.
Reasoning Director of Marketing Thomas Fry refused to reveal the identity of the commercial products that were tested in the survey. MySQL is similar to and competes with database products such as Oracle’s 9i, IBM’s DB2 and Microsoft’s SQL Server.
Fry said the study included products from some well-known commercial vendors, as well as those that are not so well known, including makers of embedded databases.
Asked what kind of conclusions Reasoning drew from the study, Frye told internetnews.com his company believed MySQL boasted fewer defects per line because proprietary software lacks the open peer review process of open-source projects such as MySQL.
Peer review, he said, enables many programmers to examine code, which often results in more flaw detections. Also, he said many users don’t just report defects, as they would do with commercial software, but actually track them down to their root causes and fix them.
Fry also indicated stringent deadlines placed on proprietary software may
force some products to the public realm before they are truly secure, or as
bug-free as possible.
Mountain View, Calif.-based Reasoning, which provides code-review services to technology clients, said it compared MySQL v. 4.016 to several proprietary database products, finding that the “defect density” — defined as the number of defects found per thousand lines of source code — was greater in products that are shipped and sold by companies.
For example, Reasoning found 21 software defects in 236,000 lines of MySQL source code. The defect density of the MySQL code was 0.09 defects per thousand lines of source code. Reasoning, which scrutinized over 35 million lines of commercial code, found that the commercial average defect density of these projects came to 0.57 defects per thousand lines of source code.
IBM had no comment for this story. A Microsoft spokesperson expressed
doubt that SQL Server was examined, and wondered how it was possible unless Reasoning had gotten its hands on its SQL Server code. Analyst Carl Olofson, who researches the database software market for IDC, said the criticism of Reasoning’s methods is valid.
Fry said Reasoning’s clients submitted source code as part of the software inspection services the company provides.
MySQL AB Co-founder and Vice President David Axmark, whose company’s
database benefited from the study because Reasoning helped his developers
pick out defects in the software that had gone unattended, said the findings
validate the open-source development method. MySQL will release a new
version of its database software reflecting those changes this week.
“Reasoning’s conclusion that the MySQL database software quality is
significantly higher than proprietary code validates the Open Source
development method, in which large communities of programmers ‘battle test’
the software,” Axmark said in a public statement.
But IDC’s Olofson said peer review testing comparisons between
MySQL and commercial products may not be entirely balanced.
“It should be borne in mind that the leading RDBMS [relational database
management system] products are probably huge in terms of source code
compared with MySQL,” Olofson told internetnews.com. “I suspect that
most commercial proprietary RDBMS products that have been available for ten
years or more (as is the case with all the leading ones) are pretty solid in
terms of the core functionality that MySQL offers, and that their defect
rates for just that functionality would be much lower than cited for the
proprietary products overall.”
Moreover, he said that because MySQL’s development is guided by a strategy
managed by the company MySQL AB, which owns the copyright on the code and
trademark on the name, the company is more likely to address strategic needs
of the marketplace than other open-source products.
While the open-source results for MySQL compared to commercial software were
overwhelmingly positive, Fry said some open-source products fall short
because the feature sets are not quite as advanced as the bells and whistles
found in proprietary software. This is because commercial vendors are driven
by competition to put the most advanced software they can out to the market.
Open-source developers tend to be a bit more basic in the projects they
undertake, he said.
Olofson agreed.
“I would also suggest that MySQL is considerably simpler, not having to
address the myriad features and backward compatibility issues that the
proprietary products have had to do, which almost ensures that the code will
have fewer defects,” Olofson said. “As open source products become more
complex and address more diverse requirements, their defect rate is likely
to go up.”
