Shortly after the release of Apple’s Wi-Fi-powered eye candy, the infamous iPhone, reports surfaced of disruptions on Duke University’s WLAN. After 10 days of industry speculation and furious troubleshooting, the culprit has now been fully identified.
According to Duke, “a Cisco-based network issue” caused “some minor and temporary disruptions in service.” Specifically, the way in which a tiny handful of iPhones interacted with Duke’s WLAN sparked a number of short but intense Address Resolution Protocol (ARP) storms. While each storm lasted just 10-15 minutes, they effectively prevented WLAN service delivery by over two dozen APs during each interval.
In a July 20th press release, Duke CIO Tracy Futhey reported that “Cisco has provided a fix that has been applied to Duke’s network and there have been no recurrences of the problem since. We are working diligently to fully characterize the issue and will have additional information as soon as possible. Earlier reports that this was a problem with the iPhone in particular have proved to be inaccurate.”
But of course other large organizations with Cisco-based WLANs remained anxious to understand both the cause and the fix.
On July 24th, Cisco published a security advisory that detailed the root cause of Duke’s highly-publicized incident. Several Cisco Wireless LAN Controller (WLC) products were found to contain vulnerabilities in the way that they process unicast ARP packets. According to the advisory:
“A vulnerable WLC may mishandle unicast ARP requests from a wireless client leading to an ARP storm. In order for the vulnerability to be exposed, two WLCs attached to the same set of Layer-2 VLANs must each have a context for the wireless client. This can occur after a Layer-3 (cross-subnet) roam or when guest WLAN (auto-anchor) is in use.
- If the client sends a unicast ARP request with a destination MAC address [not yet] learned by the Layer-2 infrastructure, that request will be flooded to all ports in the Layer-2 domain after egressing the WLC. This allows the second WLC to reprocess the ARP request and incorrectly reforward the packet back into the network.
- If the arpunicast feature [is] enabled, the WLC will re-forward broadcast ARP packets targeting the IP address of a known client context. This creates an ARP storm if more than one WLC is installed on the corresponding VLAN.
- In a Layer-3 roam, [when] a wireless client moves from one controller to another where the WLAN interfaces configured on different controllers are on different IP subnets, a unicast ARP may not [be] tunneled back to the anchor controller, but may instead be sent by the foreign controller to a local VLAN.”
Cisco has issued a fix for the affected products, which include Cisco 4000, 4100, and 4400 Series Wireless LAN Controllers, Cisco Catalyst 6500 Series Wireless Services Module (WiSM), and Cisco Catalyst 3750 Series Integrated Wireless LAN Controllers.
As a workaround, Cisco recommends forcing all wireless clients to obtain local IP addresses from a DHCP server, using the DHCP Required setting. This simply prevents clients from attempting the standard steps (see RFC 4436) that apparently triggered Duke’s incident. However, a malicious client can still intentionally cause an ARP storm.
The iPhone’s role in all of this? This high-profile Wi-Fi client aimed a huge spotlight on what may have otherwise been a relatively typical vulnerability discovery and fix. Organizations prepping for the expected onslaught of iPhones can breathe a sigh of relief. But don’t be completely surprised if other enterprise WLAN vendors quietly find and fix similar vulnerabilities. As always, keep an eye on security advisories for your own gear.