Study: MySQL Hard on Defects
Page 1 of 1
An unpublished study from code analysis firm Coverity indicates that open source database MySQL has relatively few defects in comparison to its commercial competition.
Coverity analyzed MySQL 4.1.8 last month and found a total of 97 bugs among the open source database's approximately 425,000 lines of code. The defects cross over nine different categories, including memory/resource leaks, various forms of nulls and overrun conditions (including Deadcode, forward null, negative returns, static over run, resource leak, reverse null, unused value, uninitialized variable, use after free and unused value).
"Often times in a database we find memory leak problems and resource leak problems," Seth Hallem, CEO of Coverity, told internetnews.com. "We didn't really see much of that in MySQL, which is a good indication they've tested it thoroughly, and that they've designed the code well to avoid those types of problems."
Additionally, Coverity's study found that overrun conditions comprised only five of the total 97 errors. Buffer overruns have been commonly used as an attack vector by hackers on a variety of applications, though according to Hallem, they're generally not the most common form of software defect. The Coverity CEO noted that roughly 80 percent of software bugs are null pointer bugs.
"Overruns are perceived as a greater threat because malicious attacks involve injecting code as a result of overrun, and the effects are more disastrous," Hallem explained. "Null pointer may only result in intermittent downtime and until some hacker figures out how to trigger it constantly you'd get a denial of service. I think people trigger them accidentally all the time."
Coverity's study did not analyze MySQL on a particular production platform. In fact it didn't directly test MySQL on any individual production platform at all. Instead its analysis looked at the compiled code base, which in Coverity's view gives them insight into all production environments.
"We explore all paths to the code," Hallem said. "So in essence we're looking at all and any production environments at all times. And because we're looking at all the paths, the production environment may manifest itself as different pathways or behaviors in the code. We're trying to look at all possible behaviors, so in a sense we're analyzing all production environments."
That said, as evidenced by last weeks attack on Windows versions of MySQL servers, certain environments are more susceptible to attack than others.
"The effects of a security flaw in MySQL and also in many other open source products are more disastrous on Windows, because Windows does not have the same kinds of operating system protection available," Hallem explained. "That said you're also putting a lot of trust in the hands of the administrator. A MySQL admin can always make a mistake and give too much privilege to a MySQL process on Linux. It's just that by default it's much safer on Linux."
Though Coverity's study found 97 flaws, that doesn't necessarily mean a hacker had that number of opportunities to compromise or otherwise attack a MySQL server.
"What it means is that a certain subset of those are actually triggerable by the outside world. Many of them may be triggered by out-of-memory conditions or configuration issues, or there are many different possibilities," Hallem said. "There are some of them and at least one that we've confirmed that could have been triggered by an outside user, and it was fixed very quickly by MySQL."
In fact according to Coverity, most of the defects discovered would not be considered major, though MySQL has now made all necessary fixes. But that doesn't mean that MySQL is to be considered 100 percent bug free.
"What it means is that the flaws that our tool can detect have been fixed, which undoubtedly improves the quality of code," Hallem said. "We can't make any guarantee or any statement as to whether there are any other bugs that are outside of the scope of what our tool can do. Our tool does a great job of finding certain kinds of bugs, but we cannot find everything. "
According to Zack Urlocker, MySQL vice president of marketing, the results show that MySQL has one-quarter the level of defects or issues compared with equivalent commercial software. He also noted that a similar result came from a different study last year, which showed that MySQL had one-sixth the defect level of equivalent closed source software.
Urlocker said that the commercial vendors against which the comparisons are made do not publicize their error rates, so there are no apples-to-apples comparisons between closed and open source offerings.
"When you can compile your code and not have to show your source code to anybody, that's very different than when you are publishing your source code to your peers and to the whole world, and everyone can see what you've done," Urlocker told internetnews.com. "That kind of peer review in the open source world has helped to ensure that the standards by which open source software is developed are extremely high."