 |
Terry Childs, the system administrator who is in jail awaiting trial for, in effect, holding San Francisco’s fiber-optic wide area network hostage back in July, continues to darken the lives of members of the city’s IT department.
Childs had installed equipment on the network without authorization and essentially taken it over, creating a super password, then refusing to hand it over until the city’s mayor, Gavin Newsom, visited him in jail a week after his arrest. Then, on Aug. 28, the IT department got a shock: It found yet another unauthorized device on the network.
That was a terminal server, and “it was probably pulled immediately,” Ron Vinson, chief administrative officer and deputy director of the San Francisco Department of Technology, told InternetNews.com.
The department is now scrutinizing the network even more closely in fears of getting yet another unpleasant surprise. “We don’t believe we’ve found all the devices, so we’re going to continue going through the network,” Vinson said. “Just this morning they came into my office and went through all the devices there,” he added.
His department is working with high-tech consultants Xtech, a minority/women business enterprise joint venture between two San Francisco-based companies that has a contract with the city and county of San Francisco for all technology hardware, software and services procurement. Xtech is partnering with Cisco, (NASDAQ: CSCO), which provided the networking infrastructure, to help with the remediation, Vinson said.
Why did a trusted systems administrator such as Childs suddenly turn rogue? The fiber-optic WAN he was working with connects all of San Francisco’s computers, handles city e-mail, payroll and other functions and also handles some of the systems of the city’s police department, and it would make sense to only provide access to a critical network like that to someone who can be trusted.
“When you get levels of access to things in the city, there’s protocols to be followed,” Vinson said. “If it’s anything to do with the police and fire departments, you may need to have specific background checks,” he said. “The computer department currently doesn’t have these protocols in place.”
Failed processes
It’s more than just a lack of protocols; the city’s processes and systems are in disarray. Childs, 43, had been convicted twice of aggravated robbery as a teenager and of misdemeanor weapons possession in 1995, when he was 30 according to the San Francisco Chronicle, facts that should have shown up on the employment application anyone applying for a job with the city has to fill in.
Apparently the process failed somehow, and he was hired in March 2003 by the City Department of Telecommunications and Information Services, now known as the Department of Technology, as a network engineer, the San Francisco Chronicle said.
Childs only came under suspicion earlier this year when the Department of Technology began beefing up security after getting funding from the city government. “We had hired a new security chief and were conducting inventory before implementing new security protocols for the network, and at that point certain things were discovered that looked to be suspicious,” Vinson said.
In May, Child’s managers found he had filled a room in the department’s Market Street offices with computer equipment nobody knew anything about, the San Francisco Chronicle said. They also realized Childs controlled access to the city network.
The rogue devices linked to the network were not discovered earlier because the San Francisco IT department’s change-management system is manual, not automatic. “When someone makes a change, like conducting maintenance on the network, it’s his job to put in that this is happening and it gets out to the stakeholders who are affected,” Vinson explained. If that change isn’t put in, “another system may pop up and say this system went down.”
San Francisco’s asset discovery and management processes were also antiquated, so Childs was able to work around them. The city is updating them now.
Once management found the roomful of equipment and realized Childs had sole control of the city network, it launched a background check, and “we discovered that Mr. Childs shouldn’t have had access to the police network because of his prior history,” Vinson said. Childs had a confrontation with colleagues in June, was reassigned and told to surrender the passwords and usernames for the network in July, and ended up being arrested after he refused.
That is doing things in reverse, Lew Smith, product manager for virtualization solutions at Interphase Systems, said. “When you look to bring individuals onto your IT team, make sure you have a really good screening process,” he said. Also, ensuring redundancy between key players for cross-checking is key. “Having multiple individuals with similar roles would help prevent something like this,” Smith said.
Next page: It’s all about money
Page 2 of 2
It’s all about money
Childs had been hired by a previous chief operations officer, Dana Hom, now a computer consultant, the San Francisco Chronicle said, so the IT department had no control over his hire. And money was tight, so staff were heavily worked and it was difficult to ensure staff redundancy as Smith suggested.
“It’s been difficult for the past few years, and we’re glad for the money we were able to get based on the economy and the budget shortfalls the city and county have faced over the years,” Vinson said. “We never designed the system for one person to have sole access,” he added.
The city’s network engineers are rotated from project to project, but Childs somehow got rotated less than the others. This will change. “We’re going to make sure that there’s the necessary checks and balances to make sure no one person has sole access,” Vinson said.
Vinson could not say when the system will be fully functional because “we’re still going through remediation.” Despite this, “We feel confident that we have control of our fiber-optic network,” he averred.
Childs appears in court again Sept. 24. His case is being handled by Assistant District Attorney Conrad Del Rosario of the San Francisco DA office’s special prosecutions division. Erica Derryck, deputy communications director of the DA’s office, declined to say anything because “this is a pending case, and I’m not at liberty to comment on any pending case.”
