There are two things that are obvious about disaster recovery (DR) testing: everybody with a DR plan should test,...
and most don't test frequently enough.
During a time when IT people are often asked to do more with less, full-blown DR testing can fall by the wayside although administrators may target key pieces of their plans.
Surveys make the first point clear. Symantec Corp.'s annual disaster recovery survey this year found 35% of respondents worldwide with a DR plan do not test it more than once a year, and that number is actually a 12% improvement from 2008. Lack of IT staff time was the top reason given for not testing more, with 48% pointing to that. Disruption to employees, budget restraints, and disruption to customers were all mentioned by at least 40% of respondents.
In a recent Storage magazine/SearchDisasterRecovery.com survey, 59% of respondents said they test disaster recovery plans regularly but 28% test once a year or less. Most of the blame for not testing went to lack of a DR site, and inadequate staffing and money in that survey.
Admins search for right time to test
Troy Downing, systems analyst at Rain and Hail Crop Insurance Co. in Des Moines, Iowa, says the chief challenge for his two-person storage team is shutting systems down in a business where companies expect 24/7 service. Downing said testing typically occurs when new equipment or major changes are introduced into his IT shop.
"We bought some new Sun servers in the past year, and we go ahead and test the backup on those to make sure our old procedures are up to date, documented, and apply to the new product," he said.
He says he expects to conduct annual widespread disaster recovery testing of every application when Rain and Hail completes work on a new DR project, which includes VMware at the DR site.
"We're more likely to do that in our new DR environment," he said of full testing. "VMware allows for site recovery tests. You can get as close to a live test without doing a full test."
Eric Hess, also one of a two-person storage management team at the College of American Pathologists in Northfield, Ill., said a large archiving and virtualization project underway in his shop reduced his two usual scheduled DR tests this year to a "mini-test" of key applications. "We've been so busy with the virtualization project," he explained. "It's a time and resources thing."
Experts advise to test DR plans at all costs
Consultants and analysts who specialize in data protection recommend DR testing despite the hardships. SearchDataBackup.com executive editor and independent backup consultant W. Curtis Preston suggests that companies do a large scale disaster recovery test once a year, with smaller, alternating tests throughout the rest of the year. While he acknowledges that testing isn't easy, he doesn't have much sympathy for those who refuse to test their DR plans.
"It's disruptive; it costs a lot of money," he said. "But if you don't? Well then you're prepared to take that risk to what could happen."
Preston has several suggestions to save money. He recommends testing individual aspects of the disaster recovery plan as opposed to the entire thing. He acknowledges that is a challenge for companies running a 24/7 platform, but worthwhile in the end. For companies that close on weekends, he suggests testing on a Saturday afternoon when the risks are significantly less.
Taneja Group senior analyst Jeff Boles says the Symantec survey numbers don't surprise him, but he's encouraged by the 12% jump over last year. He says that in past years, data recovery was "frequently the last item on the budget approval process."
He noted that organizations "could take some reassurance that there was some measure of DR there, as a checklist item, and not really think strategically about it. I think with our increased focus on service levels and service level management, organizations are more aware today of the strategic importance of DR preparedness and they try to incrementally improve it on a regular basis."