You think you have network problems? Try this one on for size. You’re a network administrator at the University of Wisconsin (UW) in June when suddenly your Network Time Protocol (NTP) server is getting buried alive by NTP requests, to the tune of 250,000 requests per second. That’s hundreds of megabits per second, more then enough to kill any server this side of a supercomputer and enough to put a sizable dent into any university’s bandwidth pipeline.
And, who were the villains behind this dastardly distributed denial of service (DDoS) attack? Iraqi sympathizers? Bored hackers? Minnesotans?
No, it was Netgear.
Specifically, it was Netgear routers — models DG814, RP614, RP614v2, MR814 and HR314 (the latter two use Wi-Fi). Hundreds of them. Thousands of them. Indeed, the UW estimates, hundreds of thousands of them — all asking for the time.
These routers had one little problem. Like many routers they need to keep accurate time for logging purposes. And, like many routers, they automatically run an NTP client to reset their clocks. Where things went wrong was that 1) all were set to use the NTP server at UW for their time-setting and 2) immediately requested the time again if they were unsuccessful in getting the corrected time, the first time, or the second time, or the you get the idea.
With this setup, it was only a matter of time (don’t pardon the pun) before enough Netgear routers all tried to get the time at once, failed, retried, and in the meantime other Netgear routers came crowding up and also started to fail and retry. Before the UW knew it, the server was dead as a doornail, the network was getting buried and there must have been an exciting time indeed at the school’s network operations center (NOC).
Annie Stunden, chief information officer at UW, isn’t too unhappy about the inadvertent DoS attack. Now.
Netgear, realizing what their equipment had done, and knowing a public relations disaster when they saw one, is providing cash for UW to build out its network to deal with future DoS attacks from out of date Netgear routers.
There is a firmware fix for these routers, but these routers are all for home use. Home users being home users, Netgear expects many of them to never be aware of the need for a firmware upgrade — after all, to them the router seems to work just fine. From where a home user sits, a constant stream of NTP requests takes up so little bandwidth that it’s quite unnoticeable. It’s only when hundreds of thousands of routers storm an NTP server’s virtual gates that it becomes a problem.
Another problem, of course, is that firmware upgrades are never easy to do. It’s likely that Joe home-user, faced with a difficult update for a problem that as far as he can tell doesn’t affect him in the least, may not apply the update. Besides, as we all know from Blaster and the like, even when a patch is easy to do and well publicized, many home users still won’t apply it.
The Netgear patch does two things. The first is that sets the NTP client to take longer and more truly random intervals to get the time again after it fails to get the time. The other is that Netgear is now running its own NTP servers and pointing the clients to try their servers before asking anyone else for the time.
This is a rather shocking problem. It’s not like there’s anything new about NTP clients, and it’s basic programming practice to set them, like any network service, to re-check the servers at random intervals. In addition, such popular NTP PC clients like the shareware Windows program Tardis commonly automatically move from one NTP server to another if at first they fail to get the time at one.
Still, mistakes do happen and at least this one has been corrected. And, for the sake of NTP servers everywhere, let’s hope this one is the last one.
