That was close. The State of California on Tuesday headed off a near-disastrous shutdown of its top-level domain that would have taken all of the state’s government Web sites offline.
The problem began in Marin County, north of San Francisco. Hackers apparently compromised the county transportation authority’s Web site on Tuesday, redirecting traffic to pornographic sites.
In response, someone at the U.S. General Services Administration (GSA), the federal government’s property management agency, “saw some inappropriate activity with Web access being redirected and removed .ca.gov from the root directory,” Jim Hanacek, spokesperson for California Department of Technology Services (CDTS), told InternetNews.com. The GSA’s duties include responsibility for all the of the .gov sites, of which .ca.gov is a sub-domain.
Hanacek credits the GSA with doing its job in identifying there was a problem. But he said the agency shut down California’s domain apparently without first consulting any authorities in the state.
“It was like using a shotgun to kill a flea,” he said.
Once CDTS officials recognized the problem, they promptly launched an emergency operations center meeting, just as they might do if a natural disaster had disabled part of the electrical power grid. Hanacek said he isn’t sure who at GSA initiated the shutdown, but once his group contacted top officials at the federal agency, they handled the issue promptly and reversed the action.
Because officials identified the problem and responded quickly, California only suffered some isolated incidents of e-mail and government networks becoming inaccessible, Hanacek said. The affected services were back online in “about two hours,” he added.
“They did what’s called a forced propagation instead of a scheduled update,” he said. “So the update to restore .ca.gov happened immediately and everything came back the way it was supposed to be.”
If there’s a silver lining to the story, it’s that the state and government departments have started a dialog, hopefully to avoid such problems in the future.
“Sometimes system administrators are the kings of their domains and they take care of things from a technical perspective that might not be the right decision,” said Hanacek, reiterating that he isn’t sure who initiated the shutdown. “I’m sure we’ll now be putting processes in place going forward that these kind of actions aren’t done without prior
notification.”
The statewide shutdown came close to prompting a call by Governor Arnold Schwarzenegger to the President, according to a Marin County transportation
official quoted in the Marin Independent Journal. The paper said the
shutdown began at 4 p.m. Tuesday and the network was back up by 7:30 p.m.
A pornography ring based in Eastern Europe evidently is to blame for the initial computer infiltration at the Marin County Transportation Authority, according to the Independent Journal report, adding that a computer consultant traced the attack to the group, which apparently had figured out how to use the Marin Web site’s excess capacity as a portal for porn.
County officials quoted in the report said they first discovered the problem in
April and beefed up its security only to see the problem reappear last month.
At any rate, the hackers’ target for distributing porn was certainly an odd one — a site that mainly distributes information about Marin’s transportation issues.
If nothing else, Hanacek said the incident confirms his belief that in today’s complex data centers, “you can’t anticipate everything that can go wrong … But we need a process to get systems back as quickly as possible.”
