Hurricane Sandy missed Strand Associates' Midwest offices, but the engineering firm was prepared for the worst,...
thanks to a disaster recovery plan that survived Hurricane Ike a few years back, and a regular DR test.
Strand's storage architect Justin Bell said he recognized the same problems in East Coast companies hit by Sandy as his firm experienced when Ike took out one of its Ohio offices in 2008.
"With Sandy, we saw some places were flooded, but the vast majority of sites had problems because they were without power for so long," Bell said. "The major impact was lack of power."
Bell said Strand's disaster recovery (DR) plan focuses mainly on surviving a loss of power at any of the company's 10 sites. Strand's headquarters is in Madison, Wis., with two remote sites each in Ohio, Indiana and Kentucky, and other remote sites in Milwaukee, Wis., Joliet, Ill. and Phoenix, Ariz.
Strand has direct-attached storage at each remote site and a Dell Compellent storage area network at headquarters. It uses FalconStor IPStor 7 and continuous data protection (CDP) to protect data and failover between sites. Strand backs up data in each remote office, and then replicates it to headquarters. Data from headquarters gets replicated to the Joliet remote site. The firm also sends data off to tape once a week as an extra precaution.
"We've always required that we have two copies of data on-site for each remote office, and then a remote copy in a distant geographical location to provide for DR," Bell said.
When Hurricane Ike hit, Strand lost power in one of its Ohio offices for a week. Bell said users in other offices had been accessing data stored at the affected office and didn't even notice the office was offline.
"We mounted up a remote copy of that data to our Madison office, and we used our directory structure and DFS [Windows distributed file system] to make it appear that all their files were in the exact location to our users," he said. "We transparently put the data back to where it needed to be from the user perspective without disrupting any projects we were working on."
"We try to focus on things we determine to be the most likely to happen," he said. "We've done practice for a fire in a remote office or headquarters. We've also tested for an extended power outage because that's the most likely scenario that would cost us the biggest headache."
In its DR test, the company may pick one office to test how it can survive a loss of power.
"It's important for us to have a clear outline for what type of communication has to go out to our users, and to define the roles for all of our staff in the event of a power outage," he said. "We've created separated roles for what needs to be done in the event of a power outage. We've laid out instructions clearly so those roles can be handed out to whoever is available and we can take care of whatever processes we need to go shut down equipment correctly and take care of our users."
If headquarters goes out, its servers would roll over to Joliet. "It's far enough away that we're comfortable that it wouldn't get hit by the same disaster, but close enough that we can drive there in three-and-half hours and pick up where we left off," Bell said.
Bell said he chose FalconStor's software because of its CDP and data deduplication features. The CDP reduces Strand's recovery point by making a copy of data every two hours, and the dedupe reduces the amount of bandwidth needed to replicate data. He said Strand previously used a combination of products for failover, but found it's more efficient to have one vendor during a disaster.
"In the event of a disaster, you don't want pieces from different companies," he said. "As annoying as finger-pointing is normally, it's not acceptable during a disaster."
Bell said his data protection plan also helps with smaller problems, such as when servers go down and require a few hours to replace, or individual files get deleted or corrupted.
"By having a second copy of local data, as well as a remote copy, you can fail back to the second copy and present users with the same level of service as before the server went out," he said.
"DR doesn't provide a company any benefit unless you have a disaster. You gamble on whether you're going to lose more money if a disaster happens, or if you will spend more money preparing for a disaster that doesn't happen. If you can provide other business benefits, that tilts the equation.
"We get requests every day for restores because our engineers know we can save them time restoring a file that was a problem. Instead of going back to recreate what they deleted, they can just call us up and we can get it back for them in one to two minutes -- and that's either a file from an hour ago or a year ago."