Crawford Adjusters Canada had a number of reasons to go looking for a new backup and recovery solution.
For starters, the company’s aging backup process wasn’t doing the trick anymore, and more and more clients were requiring proof of a solid disaster recovery plan.
Also, as a wholly owned subsidiary of Crawford & Company, a Georgia-based independent company and one of the largest providers of claims management solutions, Crawford Adjusters Canada also had to comply with the U.S. Sarbanes-Oxley Act (SOX).
Those factors drove the company to overhaul its backup and recovery infrastructure, but the change had other benefits too. For Crawford Adjusters, getting its backup processes and disaster recovery strategy in order resulted in IT efficiencies, easier administration and cost savings.
The Ontario-based insurance services provider, a claims adjuster, has 85 branch offices and 900 employees. Over a number of years, Crawford Adjusters CIO Scott Sutherland saw a couple of IT issues converging: one was a deteriorating IT backup process, and the other the need to improve the company’s disaster recovery solution.
“While searching for a full disaster recovery solution, we found Asigra, whose products we use for both disaster recovery and to help us improve our backup process,” says Sutherland.
Asigra Inc. provides agentless distributed backup and recovery software for network computing.
Today, Crawford protects its server and workstation data using Asigra’s Televaulting DS Server solution. Asigra Televaulting is a WAN platform solution that delivers backup/restore service as a utility.
Going Downhill
According to Sutherland, a slow deterioration of backup services had been occurring for a number of years. The company’s 85 branches are served by NT Servers with file and print functions. The locations have as few as two and as many as 100 employees. There are 25 server-class servers and 65 workstation-type servers in all. The largest branch has four servers, other locations have fewer. The company had 30 older DLT drives scattered in the field which failed periodically, sometimes resulting in no backup services at all.
Branch office data are claims-related Word and Excel files, digital camera pictures, and a couple of databases at the larger offices where scanned images are stored. There are approximately 3.5 terabytes of storage capacity at the branch level, with about 2.5 terabytes being used.
One upgrade option considered by Sutherland was to purchase new tape drives. However, that solution had drawbacks: cost; the need for someone to manage the drives; and the issue of taking them off-site everyday.
“Employees at the branches aren’t IT-savvy, yet were responsible for backup,” he says, noting that simply installing new drives didn’t address the administrative issues.
While aware of the need for a new backup solution, Sutherland was even more pressed to find a new disaster recovery solution. Why? In the fall of 2004, Crawford had the business opportunity to be the sole vendor of insurance services for a particular customer, but the customer required submission of Crawford’s disaster recovery plan. That’s what prompted the company to improve its existing solution.
“Since September 11, more customer RFPs required submission of disaster recovery plans,” notes Sutherland.
At the time, the company had an off-site location where they replicated a SQL database on an IBM HS40 Quad Xeon Server in real-time. The problem? “The off-site location wasn’t very secure and the SQL replication was prone to failure,” he says. In short, a lot of administrative overhead was required to keep things going.
The system being replicated was an in-house developed, client-server-based core claims system that resided on a SQL server. The system was centrally located and had up to 700 concurrent users.
Time for a Change
Sutherland drew up a requirements list for a new disaster recovery solution: the company needed an off-site location, or data fortress, where the replication of core critical services was continuous even if a major disaster hit the central IT center; the ability to switch the company’s Internet presence to another location; the ability of employees to keep working; and e-mail service had to be maintained.
Before finding a consulting partner, Crawford’s IT department did a risk assessment of its core applications and determined the critical timeframe that applications had to be up and running.
Then, as luck would have it, Sutherland heard from a former business partner whose new business, E-ternity Business Continuity Consultants, was exactly what he needed.
In January 2005, the work to find a new disaster recovery solution was underway. By July, after the consultant helped design and provide the specifications for a solution, Asigra came into the picture. The new solution would address both disaster recovery and backup.
Crawford, who already had an IBM and EMC SAN, brought in the Asigra Televaulting product, increased its SAN capacity, and purchased VMware and EMC’s RepliStor product.
According to Sutherland, the benefits of the Asigra solution for backup were twofold: no software was installed at the branch locations, and, once an initial snapshot of the server was taken, all that was required were nightly backups which were done on the binary level and addressed changes only.
“We removed the responsibility of backup from the branch offices and centralized it all to one tape session,” he says.
Moving Ahead
Prior to rolling out the solution, Sutherland and his consultants ran a test at one of the branch offices. The test involved simulating a failure and recreating 20GB of data. “We like the fact that we’re able to retrieve a file versus the entire catalog,” he says.
Sutherland’s IT team and the business continuity consultants worked together during the initial phase of the rollout to a couple of branch locations. Then Crawford’s IT team took over the installation of the Asigra agentless client software on one node at each remote location. Initial backups at the remote locations took between four hours and 20 hours depending on the size of the location. Over a two-week period, all 85 branches were migrated to the new solution.
“Today, all nightly backups are completed in a two hour window,” says Sutherland.
Crawford initially purchased a 200GB license from Asigra to address the remote locations. Then Sutherland undertook a server consolidation project at the main location — consolidating 20 Intel servers to 14 blades, two running VMware on an IBM blade server — and made an additional purchase of 800GB from Asigra, for a total of 1TB, which addressed backing up the 14 blades in the server room.
“We can rewrite the server room in a four- to five-hour window,” he says.
EMC’s RepliStor, handles real-time replication of the core systems at the SAN level, basically creating a hot copy of the Asigra store. The company has about 4 terabytes of SAN storage in the server room being used by the blade servers.
The Asigra store will be replicated off site. The project is scheduled for completion by the end of February.
An additional project, expected to begin in March, will be the installation of EMC’s Centera, a content addressed storage (CAS) solution, which will be configured with four separate internal SAN units and 2TB of capacity, for off-site document and e-mail storage and retrieval.
“Centera will give us a quick archival solution,” says Sutherland.
Big Benefits
Centralization of the backup function has lifted a huge burden at Crawford. “We now have assurances that backup occurs as it should with minimal administration,” says Sutherland. Other benefits include real-time notification and the ability to restore a single document or file quickly. A hot site can also be restored directly from disk.
“We also back up the entire machine and can point Asigra to restore a machine on a new machine, including the operating system and data,” he adds.
Finally, says Sutherland, Crawford’s new disaster recovery solution “meets the company’s high standards for compliance, security and ease of use.”
For more storage features, visit Enterprise Storage Forum Special Reports
