RealTime IT News

ISPs Among First to Detect, React to Outage

At 4:10 p.m. yesterday, techs manning America Online's control center saw the number of members using its service plummet from 2.4 million to 2.1 million in seconds.

"There was a noticeable and precipitous dip," AOL spokesman Nicholas Graham told internetnews.com. "The lower numbers remained throughout the evening and early morning hours."

At 11 a.m. today, about 1.9 million AOL members were online, still about 200,000 short of last Friday at the same time.

Throughout the United States, ISPs and network operators were among the first to detect an infrastructure problem thanks to sophisticated monitoring systems. In addition to a New York drop, carriers saw traffic elsewhere surge as users logged onto news sites.

Graham said AOL has made significant investments in its own systems, adding redundancy and capacity. But that didn't help customers without power for their PCs and laptops and batteries last only so long.

Some AOL dial-up numbers were knocked out when phone systems died, however most users could select another AOL access number, Graham said. Some AOL modem banks were switched to backup power.

Like in AOL's Dulles, Va., center, Akamai's Cambridge, Mass., network operations center also saw the disturbance.

The facility resembles a NASA mission control room, with giant flat-screen displays mapping Internet traffic flow, drawing data from 14,000 content delivery servers in 1,100 networks in 70 countries.

It had already been a busy week. Akamai was watching the Blaster worm that infected hundreds of thousands of PCs and laptops when the blackout hit. Companies and individuals were online all week downloading patches.

"It looks like (the Internet backbone) performed reasonably well," said Andy Ellis, Akamai's chief security architect. "There were minor spots went down because they were without enough backup power. But most (carriers) were able to route traffic around the problems."

Akamai saw significant traffic for its media customers, which include CNN and the Washington Post's Web sites.

Companies and carriers that put a renewed emphasis on backup and recovery since 9/11 probably helped lessen the impact of the blackout, Ellis said.