What do you call a wake-up call for a wake-up call?
Whatever you call it, officials responsible for securing and maintaining Marin County’s (California) transportation authority Web site slept right through it for more than three weeks, leading to a cascading series of events that culminated Tuesday afternoon with a federal shutdown of the state government’s Internet and e-mail service.
Hackers, eventually identified as porn peddlers based in Eastern Europe, managed to weasel their way into the site’s DNS server (DNS)
When someone at the U.S. General Services Administration (GSA) noticed the problem Tuesday morning, the agency yanked the entire ca.gov sub-domain from the root directory. Around noon in Sacramento, a staffer at the California Department of Technology Services (CDTS) opened the explanatory e-mail from the feds and, within a couple hours, the CDTS watched as hundreds of state Web sites were blocked, e-mails systems shut down and some degree of panic ensued.
It got so bad, the CDTS went into emergency mode, circled the wagons and everyone up to and including Governor Arnold Schwarzenegger was scrambling to get the GSA to restore the ca.gov sub-domain. Salvation finally arrived around 5 p.m. when the GSA initiated a forced propagation to update and restore ca.gov to the root directory and bring all the state’s Web sites and e-mail systems back online.
But all this drama could have been avoided had officials at the Marin County transportation authority listened and responded to several warnings it received in early September from security experts who knew the site had been hacked for months.
Alex Eckelberry, CEO of Sunbelt Software, a provider of security software based in Clearwater, Fla., personally sent an e-mail to the transportation authority on September 12, warning “you should know the TAM Web site is hosting porn and spyware.” He wasn’t alone. Others sent similar warnings through its Web site and directly to Dianne Steinhauser, the transportation authority’s executive director.
Sunbelt Software and other security experts voluntarily are constantly on the lookout for hacked government sites and constantly send out e-mails and make phone calls to alert authorities when their sites have been compromised. Sunbelt’s entire interaction with the Marin County transportation authority is documented on the company’s blog.
According to a report in the Marin Independent Journal, Steinhauser and her staff didn’t jump on the problem because they didn’t trust the warnings, thinking the repeated e-mails were probably just phishers looking for their own way to hack into the site.
Obscured by the temporary shutdown of many state Web sites, the crisis management efforts of the CDTS and the repeated warnings from concerned security experts is the fact that the agency’s Web site was hosted, maintained and apparently not secured by a third-party provider.
“There are still hundreds of compromised .gov sites out there,” said Paul Ferguson, a network architect at Trend Micro and one of many volunteers who routinely scours the Internet for security vulnerabilities at government Web sites. “These smaller, regional county government agencies outsource their Web page development and hosting because they don’t have the budget to do it themselves. The days where you could develop a Web presence, put it out there and then just forget about it are over.”
Ferguson told InternetNews.com he doesn’t want to assign blame with the government agencies that he says are constantly struggling with budget constraints and bureaucracy. However, he said, outsourcing these sites makes them sitting ducks for industrious hackers because third-party vendors often fail to do the constant patching and routine security checks required to at least give hackers pause before seeding their malware on the sites.
“It’s kind of a scary that these sites have become the low-hanging fruit for the bad guys,” he said. “The problem is just mind-boggling and most of these problems would never be noticed by the average person going to the site to see if they have to report for jury duty.”
