Like other businesses in the New Orleans area, the employees at law firm Deutsch Kerrigan & Stiles (DK&S) know they are vulnerable to hurricanes this time of year. However, they also know the firm has a battle-tested disaster recovery
The DK&S disaster recovery plan was devised after the Katrina disaster in 2005. The firm's office building was closed for about a month after Katrina. Its disaster recovery plan then consisted of a few staffers who had evacuated to Lafayette, La., coming back into New Orleans accompanied by a police SWAT team to fend off looters to retrieve servers, desktops and laptops out of the main office. They loaded the equipment into trucks and brought it back to Lafayette, but Katrina also affected Lafayette and the temporary office had no Internet access for three weeks although the accounting team could set up servers and start collecting money and paying bills.
When the New Orleans office opened about a month later, the computers were returned and set up again.
"It was a big nightmare, and opened our eyes to what we needed to do to protect our data," DK&S director of administration Don Champagne said.
Keep the servers running
The first step was the make sure the firm's 120 employees have access to their data if they had to evacuate New Orleans. Now the firm uses Neverfail Continuous Availability Suite to replicates Microsoft Exchange, SQL Server and file servers to a data center in Austin, Tex., and fails over there if the New Orleans office shuts down. "We use Citrix to access data when we're not on the network, so everybody can continue to work if we have to close New Orleans and shut down servers here," Champagne said.
Champagne said he chose Neverfail over competing products from Double-Take Software Inc. and CA XOsoft because of its failover process. "It's a far simpler process for us and our users," he said. "I don't have to have consultants available to help us do failover. We run our business on a lean IT department, but this allows them to handle everything here."
Communication is key in disaster recovery planning
The firm also has plans for keeping communications open with its attorneys and other employees after the decision is made to fail over the servers and close the office. "We have a step by step list of things we do," Champagne said. "We send the DR team -- somebody from HR management, a communications manager, two IT people, and myself go to Memphis, Tennessee. We picked Memphis because it's probably the safest large city away from hurricane area."
Champagne goes to Memphis every year to line up contractors, vendors and housing for the team, including a hotel suite for the disaster recovery team to work out of. DK&S has an office service company on call to take messages and forward voicemails to the hotel. From there, the DR team communicates with the rest of the employees -- who will likely have evacuated to safe areas -- by phone, email and through the firm's web site. DK&S uses Dell MessageOne's AlertFind notification system that sends emails and text messages to tell employees when the office is closing and opening again. Housing is set up in case the team needs to spend more than a week in Memphis.
Passing the disaster recovery test
"Now we have a completely detailed plan that we put into play for Gustav [last August]," Champagne said. "That was a true test of our system. We went to Memphis, failed over our servers to Austin, and everything worked as planned. Most of our attorneys have switched to iPhones and Windows for email. The companies we made arrangements with the last three years when we didn't do anything came through like a charm when we needed them.
"We have so many ways to communicate with staff that the only people who got left out in the cold were people evacuating to an area that got hit – for instance, Baton Rouge [Louisiana] lost power for a week. People there couldn't communicate with us. Some evacuated to a place with no power and no cell phone coverage."
The DR team spent five days in Memphis, and the New Orleans office was closed for a week.
Plans set for 2010 hurricane season
The entire law firm met in mid-June to plan for this year's hurricane season. Champagne said they reviewed the DR process, and how they should communicate if a disaster is declared. "We tell them, 'if you go to place that gets hit by hurricane, you're on your own. If you go to grandma's house and she has power, you're OK,'" he said.
The disaster recovery process from Gustav cost the firm about $500,000 in hard costs and lost business, Champagne said. Employees can take personal time to take care of themselves and their families, and the firm lost five days of billable time. Still, it was less expensive than Katrina, which Champagne said was "astronomical" in lost money -- not to mention the toll taken on the staff.