Motorola Mobility Services Platform, RF Management Edition
Price: from $ 6,500 for 50 APs
Pros: Scalable config and software management, enabler for RFMS monitoring
Cons: Jam-packed GUI, overlapping events, no per-user resource views
Motorola’s Mobility Services Platform (MSP) provides single-point “care and feeding” for distributed WLANs composed of Motorola switches, access points, and clients. This Java-based console enables centralized execution of common management tasks, including configuration, maintenance, and monitoring.
Although these operations can be aimed at individual devices, the MSP excels at performing them on tens, hundreds, or even thousands of devices, grouped by site or type. The MSP’s goal is to reduce total cost of operation for “enterprise mobility domains” by automating repetitive tasks, detecting unexpected changes, summarizing overall health, and calling attention to critical faults.
We tested the MSP RF Management Edition: a software bundle that combines MSP configuration/fault management with RFMS performance management. In this review, we focus on the MSP and its lynchpin role in Motorola’s RF Management Suite. RFMS, also sold separately, is reviewed in part 4 of this series, coming next month. (Click here for parts 1 and 2.)
An MSP license lets you manage 50 APs ($6,500), up to 2,000 APs ($35,595). However, the number of devices (switches, APs, clients) a single MSP can monitor effectively depends on server horsepower and load factors like SNMP poll interval. Originally, the MSP was sold as two appliances: Lite (up to 100 devices) and Enterprise (up to 5,000 devices). Today, Motorola also sells MSP software for RedHat, SUSE, and Debian Linux. We tested v2.9 DCP 0500 MSP software running on a 3 GHz Pentium 4 laptop VMware session.
One MSP easily managed several small sites during our review (albeit with occasional VMware glitches), but large enterprise WLANs require more than one server. In that case, a central Master MSP can provide event aggregation from and single sign-on to Subsidiary MSPs distributed throughout a mobility domain. For example, an enterprise WLAN might be segmented geographically, deploying regional MSPs to maintain devices, listen for traps, and poll data, while forwarding events to one central MSP for enterprise-wide monitoring.
MSP users are granted role-based access to resources (e.g., read-only guests, read/write mobile client admins, read/write network admins). But to our surprise, every device managed by an MSP is visible to every user account. During our tests, Motorola admins had access to our test sites and vice versa. In an enterprise-class product like this, we’d prefer to control admin access at group or site level (expected in the next release).
Availability can also be critical for large enterprises. Redundancy is accomplished by installing a cold-standby MSP with access to scheduled FTP backups generated by an active MSP. If the cold-standby cannot simply assume the failed server’s IP address, all managed devices would need to send traps to both active and cold-standby MSPs. While not as efficient as active-active HA, this scheme supplies the policies and data needed to continue non-real-time management tasks during an MSP server outage.
Portals and Portlets
In enterprise management consoles, the trick is to somehow meld power with simplicity. The MSP GUI takes a valiant stab at this goal, with room for improvement.
For example, the MSP “Home Portal” provides a wealth of summary data. From this one console, NOC staff can eyeball the entire domain by asset headcount, health status, management job status, data collection statistics, and a list of recently-discovered APs. These and many other data nuggets are presented through a plethora of “Portlets” that can be customized to fit each MSP user’s job and personal preferences.
Figure 1. MSP Home Portal. (Click image for larger version.)
Customizable summaries are nice, but the devil is in the detail. We found the Home Portal far too busy, yet its summaries were too superficial. Beneath this “eye candy” lay many other Portals and Portlets, jam-packed with functionality. We almost always clicked right past the Home Portal, heading for screens where management tasks take place—typically the Mobile Infrastructure Portal. However, the sheer number of parameters found on many Portlets can be overwhelming. In our opinion, this GUI alternates between overly simple and intensely complex and would benefit from more comfy middle ground.
To its credit, this GUI is capable of administering every device in a Motorola WLAN. (To ensure this, all WLAN firmware updates are tested with the MSP prior to release.) We used the MSP to manage Motorola Wireless Next Generation (WiNG) infrastructure—including stand-alone “fat” APs and switches with “thin” APs (in Moto-speak, access ports). The MSP can also manage Motorola/Symbol mobile units (MUs) and legacy Symbol APs (non-WiNG devices). This breadth is excellent—although it understandably stops short of third-party APs.
However, squeezing diverse devices into one console inevitably adds complexity. For example, APs and MUs are provisioned very differently, albeit through the same Portals—admins must learn which Portlets, tasks, and options apply to each. Stats collected about fat and thin APs are presented entirely differently—admins must learn where to look for each.
In short, the MSP does an admirable job of gluing together Motorola’s broad product line, but it cannot hide all the seams. Users managing diverse devices (e.g., retailers) will benefit the most from this all-under-one-roof approach, while those with relatively homogenous WLANs (e.g., carpeted offices) get more knobs than they really need.
Motorola MUs have agents that auto-register with the MSP. However, network infrastructure devices must be discovered by the MSP using SNMP v2c or v3 (for WiNG products) or the built-in Symbol Entity Mobility Manager (for legacy products). We added our test networks to the SNMP Network Discovery Portlet (below), supplying public IP addresses, port numbers, community strings, and parameters that determined how often and persistently polls were repeated.
Figure 2. SNMP Network Discovery. (Click image for larger version.)
SNMP discovery is a network management staple, and the MSP’s implementation works much as you might expect. Note that networks must be identified by IP address or range—we’d like an option to supply dynamically-addressed hostnames (increasingly common for small ROBOs).
Discovery can be auto-repeated at intervals (e.g., daily) to spot missing or new devices, including Motorola fat APs (e.g., AP-5131), switches (e.g., WS-2000), and the thin APs (e.g., AP-300) actively controlled (adopted) by those switches. A MAC ACL is checked to detect unknown fat APs only; true rogue detection falls to other suite components (WIPS and/or RFMS).
New devices can optionally be auto-mapped into specified sites, whereupon pre-defined configurations, firmware packages, and event/polling policies can be applied to them. Because the MSP is fundamentally a LAN manager, all devices are labeled by MAC address. Fortunately, a .csv file can be imported to add user-friendly asset names. Unfortunately, we had to repeat that import after each MSP restart or thin AP adoption. Getting assets names directly from each device is expected in a future release.
Organizing devices into sites
Once devices are discovered, they remain in the MSP’s asset inventory forever—or until a retired device is manually deleted. To help you sift through and manage large device populations, the MSP provides the Mobile Infrastructure Portal.
This hierarchical tree presents your entire mobility domain, organized by type, IP address, or group affiliation. Here, one can easily search, collapse, or expand tree branches to find devices of interest. Events, monitored stats, configuration status, and firmware versions can be viewed for any selected device. For example, we mapped our discovery results into two defined Sites: CS_Branch, with a single AP-5131, and CS_Lab, containing WS-2000’s and subordinate AP-300’s (below).
Figure 3. Mobile Infrastructure Portal – Site List
Multi-level branches can be created in this tree by assigning hierarchical site names (e.g., region.city.office). Before discovery can map devices into sites, sites must be imported from .csv files or added individually through the Admin Portal.
Sites control how the MSP will use FTP or TFTP to manage each device. As we shall see, many MSP tasks are conducted using SNMP and file transfer. A small network might co-locate one FTP server with the MSP. Larger networks are likely to use distributed FTP servers–for example, an FTP server hosted at each remote site, used by all devices at that particular site to store backups and supply new firmware.
Each site must be bound to an FTP server’s public IP (accessed by the MSP) and private IP (used by managed devices). To accommodate this, we poked holes through Internet firewalls to permit inbound SNMP, FTP, and HTTPS. When reconfiguring dynamic IPs became a pain, we re-pointed our sites to an FTP server on the MSP itself. Had we been deploying a production network, we would certainly have secured all MSP traffic over site-to-site VPN tunnels instead.
The Mobile Infrastructure Portal also lets you sort devices based on pre-defined or customizable groups. For example, we created a “CS” group to filter all active devices and list only a chosen few, based on current attribute values (below). These static or dynamic groups provide a flexible, easy way to “drill down” to any arbitrary collection of devices. The Infrastructure Portal can then be used to start management jobs, apply event or data profiles, view alert lists, or generate reports on the entire group.
Figure 4. Mobile Infrastructure Portal – Group ViewMaintaining software
For example, the MSP can be used to update the software or firmware running on a single device, all devices in a site, or an entire group of devices.
By selecting the Infrastructure Portal’s Software tab, one can view currently-installed firmware versions (switches, APs), software versions (MUs), and update job status. For MUs only, this tab also displays Hardware Inventory details.
Software or firmware updates can be initiated from here. It’s even possible to define one job that updates many diverse devices to minimum acceptable versions, scheduling that job to run immediately or a more convenient time. Job status then shows whether and when the update was completed, flagging any errors that may have occurred.
We do not use Motorola MUs in our lab and thus did not exercise the MSP’s Software Package Portlets. Companies that do may wish to create “Rapid Deployment Profiles” to auto-provision barcode readers. (Readers scan a barcode containing update instructions and contact a designated FTP server to download new software.)
We used the MSP’s Firmware Package Portlets to upgrade our AP and switch firmware—both for testing and to fix a bug in our WS-2000. Before installation jobs can be created, firmware packages must be created. To do so, we used the Resources Portal to import WiNG firmware files, creating Packages that bind files to applicable device models, prior firmware version(s), and target version. After a Firmware Package (or Bundle) is created, it must be uploaded to the FTP server for each site.
Once these pieces are in place, the firmware upgrade (or downgrade) can be initiated from the Infrastructure Portal’s Install Software Package Portlet. When a firmware install is started for a group of WiNG devices, the MSP contacts each device at the specified time, verifies the current firmware version, and (if requirements are satisfied) tells the device to upload the firmware from an FTP/TFTP server. If a device cannot be reached, the MSP will continue trying forever or until a specified timeout occurs.
Figure 5. Using the MSP to upgrade firmware
There are far more direct ways to upgrade one AP. This multi-step process is designed for upgrading hundreds of devices at once, ensuring consistency across sites, and circumventing common failures. We successfully upgraded operational devices with the MSP, but could not use it to fix a firmware bug that caused our WS-2000 to go AWOL. In short, automation can take you a long way—especially in very large networks—but oddball cases still require the human touch.
Ensuring configuration compliance
The MSP applies these device grouping and job dispatch techniques to manage configurations, from updating parameters to checking compliance to creating backups.
Here again, the MSP isn’t the easiest way to configure a single switch—although you can always click the MSP console’s device GUI cut-through button to perform a task directly. Rather, MSP jobs automate relatively complex workflows. Consider configuration changes made using Configuration Templates and Variance Files:
1) Use the Infrastructure Portal to select an “ideal” device and export its config.
2) Edit the config to create a full or partial configuration template.
3) Use the Resources Portal to import that customized template.
4) Use the Infrastructure Portal to select a target device and export a variance skeleton.
5) Edit the skeleton to supply values for templated parameters (e.g., _#IPADDR#_).
6) Use the Resource Portal to import the completed variance file.
7) Use the Infrastructure Portal to select device(s) to be configured and apply both the configuration template and the variance file as shown in the figure below.
Figure 6. Using Configuration Templates and Variance Files
(Click image to view larger version.)
This workflow clones the configuration of a representative device (the template) and supplies parameter values unique to each device (variances). In the final step, the MSP effectively merges the two to create new configuration files to be deployed to every device that requires a similar update. Like software jobs, configuration jobs can be scheduled, approved, monitored, canceled, and restarted as needed. To manage MSP/network load, options limit the number of updates attempted simultaneously.
It took awhile to get the hang of this workflow and the inter-relationship between all the affected Portlets and files. It’s possible to simplify the process—for example, hand-crafting a partial template and corresponding variance file. But this is best viewed as a “power tool”—i.e., overkill for simple one-off changes; indispensible for major updates.
Behind the scenes, the MSP interacts extensively with devices to carry out configuration management tasks. Gartner estimates that 90% of successful WLAN attacks occur due to misconfiguration. Those configuration changes can occur for many reasons: a well-intentioned tweak by a local admin, an accidental reset to default, or a temporary but forgotten test change. In all of these cases, detecting the change is the critical first step to taking the right course of action. That’s where MSP Configuration Compliance Monitoring comes into play (below).
Figure 7. Configuration Compliance Monitoring
(Click image for larger version.)
When this option is enabled, the MSP polls all active devices to spot changes that might otherwise go unnoticed. First, it uses SNMP to periodically get a configuration update counter. If incremented, the MSP uses SNMP to ask the device to upload its config file to the site’s FTP server. The MSP then downloads that active config file and compares it to the last MSP-saved config for that device.
If differences are detected, the MSP admin can review changes before accepting or rejecting them. Accepting over-writes the MSP’s saved config, while rejecting causes the device to be restored. Imagine trying to accomplish this check manually for just one switch—and then multiply that effort by 100. Here, it’s very clear: savings could accumulate quickly. We relied on this feature more than any other during our test.
NOC staff responsible for surveillance can also use the Mobile Infrastructure Portal to keep an eye on Health and Events.
Alerts can be viewed for one selected device or for all devices within a site or group. Alerts can also be filtered by device type, severity, and time period – for example, show me critical alerts for switches during the last hour. As in most surveillance systems, administrators can take ownership of alerts, mark them as resolved, or clean old alerts from the database. Policies can also be configured to trigger event-escalation actions, like launching a script or sending email, SMS, or SNMP trap messages (below).
Figure 8. Health, Event, and Status Monitoring
In Moto-speak, “Events” correspond to SNMP traps sent by devices, signaling incidents like deviceUp and deviceDown. “Health” alerts are triggered by data collected from the device based on Event Policies—for example, deviceDown changes Health to “Warning” while deviceUp makes no change. And then there is each device’s summary “Status,” depicted as colored dots at the top of the Infrastructure Portal.
We felt that MSP Health and Events overlapped a bit, creating “duplicate” alerts that required more effort to clear. We also ran into occasional apparent contradictions, like the “Critical” status AP with no Events or Health alerts, and the AP that remained “Good” a month after being removed from service. Confusingly, fat APs generate Health alerts and Events, while thin APs generate Events only (Health alerts are viewed at the Switch level instead). And when we applied an email action to our CS sites, we received a flurry of daily alert mail about every site known to the MSP.
In fairness, Event Policy and SNMP Trap tuning might have mitigated some of these problems—but probably not all of them. In a very large network, alert correlation becomes essential, but outputs must also be intuitive and reliable. Based on our test experiences, MSP monitoring could use more polish.
Those interested in performance management will want to read our upcoming review of RFMS, the Management Suite component dedicated to real-time and statistical RF monitoring and analysis. RFMS is tightly integrated with the MSP RF Management Edition and can be launched from any MSP-displayed Site or Event.
To do its job, RFMS relies on performance data collected by the MSP. MSP Data Collection Profiles identify sets of SNMP attributes to be periodically polled and recorded. The MSP supplies a number of canned profiles, appropriate for different kinds of devices and attributes. For example, all switches are polled for default attributes for WiNG and legacy switches, while all APs have their own set of default profiles. Some canned profiles collect “static” attributes that change infrequently, while others are used to poll “debug” attributes at a rapid pace for trouble-shooting.
Custom Data Collection Profiles can be added through the Resource Portal, and default profiles can be refined by checking/unchecking individual attributes. Once defined, profiles can be applied at various points in the Mobile Infrastructure Portal hierarchy, along with specified polling intervals. For example, the following figure shows Profiles applied at both device type and group levels.
Figure 9. Data Collection Policies
Note the profiles cannot be directly tweaked for one device—you must go to the Resource Portal and create a new profile to make changes. Like Event Collection Profiles, Data Collection Profiles can also trigger defined actions. But Data Collection triggers are based on logical statements about polled attribute values—when the condition is true, the action occurs.
Data collection obviously generates load on the polled device, the MSP, and the network in between. As such, it is prudent to spend time pruning the MSP’s canned profiles and selecting collection intervals sensible for each installation. The MSP provides plenty of raw material to work with here, but beware that these SNMP attribute lists are lengthy and not self-explanatory. Our advice: start with long collection intervals and apply “fast” debug profiles sparingly. For ad hoc debugging, use the Collect Now button instead of configuring very short intervals.
MSP Portals generally offer near-real-time displays for on-going management tasks—for example, recent job status lists, current health alerts, and last-polled attribute values. Some historic information can also be obtained using MSP Reports (below). A few canned reports are built in, but custom reports can also be created here. As for other tasks, Reports can be generated about one device, an entire site, or any group of devices.
Figure 10. MSP Reports
In our view, the MSP is a network management workhorse when used by itself. It seems to do a reliable job of automating large-scale maintenance and configuration workflows. Because the MSP provides a single-point interface to so many different devices and tasks and attributes, education and experience are required to tap full potential in these areas.
Although the MSP centralizes event monitoring and data collection, these tasks have a tendency to create too much information. Here, the MSP either delivers simple summaries or raw data—there’s little middle ground in the MSP itself. However, RFMS pick ups where MSP data collection ends—think of MSP as the collection engine, but RFMS as the dashboard where analysis outputs are viewed. Similarly, the MSP gathers WIPS alerts—presented as “traps” about the WIPS server—but WIPS must be used to drill into any security alert. In those areas, the MSP plays a centralizing role but relies on other Management Suite components to add value and “deliver the goods.”
We hope the occasional process freeze and spurious event we encountered will disappear in future releases—although frankly they could have been caused by VMware, the laptop, or MSP software. Motorola’s remote test environment gave us a bigger sandbox to see the MSP in action, but it wasn’t representative of the stable platform one would establish for a production MSP.
As the MSP matures, we also expect to see a few refinements. Most notably, the ability to define per-user resource views would help cut large networks down to size and make the MSP accessible to users with limited rights and responsibilities. But just keeping up with new Motorola infrastructure devices—in the WLAN arena and perhaps beyond—will be a fairly sizeable task. Tackling that challenge is a good reason to manage large, diverse Motorola “mobility domains” through a central unifying console like the MSP.
Lisa Phifer owns Core Competence, a consulting firm focused on business use of emerging network and security technologies. With over 25 years of experience in the NetSec industry, she has been involved in wireless product and service design, implementation, and testing since 1997.