In its simplest terms, disaster recovery (DR) planning comes down to four discrete activities. First, there's a planning process when objectives are set, project scope and schedule are defined and executive management signs off. The next phase involves data collection, where the DR team goes to the organization at large to gauge business priorities and gain access to pertinent documentation that must be included in the plan. The third step is a detailed risk management assessment so that the dollars spent in DR activities protect the most critical corporate assets. The final task is threat mitigation. Once vulnerabilities are identified, enterprise companies need the right processes and technologies to ensure business integrity.
A not-so-simple plan
Seems simple enough, right? What could go wrong in these four steps? Plenty. I recently spoke with a number of companies doing DR planning and found problems throughout the process. Because the DR plan is just too important to mess up, I'll illustrate throughout this article some lessons learned to help IT avoid and overcome typical DR pitfalls.
Problems in the primary DR planning phase tend to center on deficiencies in project organization, definition or corporate support. For example, one company claimed that part of the dilemma was that its plan suffered from a lack of granularity. An IT manager lamented, "Our DR plan is too high level. It defines which systems need protection, but doesn't dive down into the equipment. We are still struggling to relate systems to technology and aren't sure if we are protected or not."
Another common complaint heard during the beginning of a DR plan is lukewarm support from executive management. One DR planning manager told me, "Our CEO signed off on the plan, but then stepped back from it. After this, support for the plan waned to the point where DR was seen as a corporate nuisance."
Lessons Learned: First off, a DR project that isn't clearly defined at its onset will likely be fraught with problems throughout its life. Some DR planners will let early problems slide in order to move the project along. Don't succumb to this temptation, as it's far more effective to slow down and get it right the first time. Make sure that the scope and details of the project are all documented. Make schedules realistic and clear. Get agreement on the plan from all departments and parties and march ahead only when everyone is on board.
Get down to business
As for half-hearted support from the corner office, this is a certain recipe for failure. CEOs and senior management must champion the plan in a visible way. Executives must sit in on training and highlight the DR plan at corporate meetings in order to maintain the role of DR cheerleader. It is essential to communicate the plan's importance and maintain focus.
During this phase of a DR plan, the project team must define business processes and collect supporting data. Several issues creep up here. For one, some DR planners may get lost trying to understand complex business processes and trying to put them into the right context for the plan. Others may understand the processes, but are stymied by poor document management. These groups spend too much time looking for data and projects fall behind. One DR executive bemoaned this fact to me when she said, "I know that the documents exist because I've seen copies, but my people are constantly asked to track things down themselves. We should be getting support, but this is like pulling teeth."
Lessons Learned: Many of the snags described here are due to poor preparation during the project planning process. As mentioned above, the first phase of a DR plan must identify the business processes, all of the right people and any necessary documentation. A thorough effort at the beginning of the project will quickly uncover any areas where data gathering may require some detective work. The DR planning effort must also be led by senior people with the right level of experience, so they can work with business people in a peer-based relationship without getting intimidated or buried by details they don't understand.
In the risk assessment--or third phase of a DR plan--companies must identify critical assets that need protection and also uncover all of the potential threats that could interrupt operations. Both of these assignments can become problematic. One manager who had trouble prioritizing corporate assets, said, "When you talk to the business people, everything in their area is mission-critical. I don't want to be the bad guy, but we don't have the staff or funds to mirror everyone's systems."
On the other side of the coin, it may be difficult determining which threat is real and which is a stretch. One IT executive said, "It used to be we were worried about natural disasters. Now we're more and more concerned about business interruptions from things like Internet worms, terrorist attacks or cyberterrorism. We need to be comprehensive, but this creates a long list of potential problems."
Lessons Learned: Companies can't leave the DR team on its own to determine what needs protection and what doesn't. This is precisely why executive management and the board of directors need to have an active role in the DR process. The DR plan should start with some assumptions about which systems will need protection. Any changes or compromises that must be made after this should be viewed as a business--not an IT--decision. Business managers may try to strong-arm an IT project manager, but will cede to the CEO every time.
The key is to be as precise as possible in defining the likelihood of each threat to each system. For example, an Internet-facing e-commerce system is far more likely to be hacked then a finance system on its own subnet. In assessing the likelihood of each threat, make sure to investigate those where the likelihood is unknown. If you are still confused by which threats are real and which aren't, get help. Seek assistance from professional services, rather than technology vendors who may use FUD tactics to scare you into buying unneeded stuff.
Is the plan ready?
Once all the preparation, data gathering and risk assessment are done, out pops a DR plan that ensures success, right? Not so fast. Even though the DR plan seems complete, numerous problems may exist. For example, sometimes the DR plan is overly complex and only the core DR team can truly understand it. I've also heard of instances where upon completion, the DR plan remains static and doesn't accommodate changes to technical infrastructure or business processes. One DR manager told me, "I'm afraid we're a lot more vulnerable than most people believe. We haven't updated our plan since the end of our last fiscal year; we never do any testing; and we've made a number or structural changes. I hate to say it, but the DR plan was obsolete the day we said it was complete."
Lessons Learned: The "keep it simple stupid" principle absolutely applies to DR. The plan should be written in a manner that is easily understood by the entire organization. Remember, people come and go; the DR plan is about the business, not the people on the DR team.
Another reality about the DR plan is that it's a living document. Changes to the business or technology infrastructure must trigger a parallel change to the DR plan. This may mean going back to step one (process planning) and progressing through the entire planning process again. But these steps may be necessary to ensure protection.
One final note--it isn't worth the time, money and effort to put a DR plan together unless the company is willing to invest in comprehensive training and frequent testing. This will maximize preparation while providing a method to uncover and fix any weaknesses.
DR plans are serious business, but are still subject to human frailties--problems can crop up at any time. If a company puts in a lot of work upfront and is then willing to work cooperatively and strive for constant improvement, the planning process will run smoothly and the DR plan will provide the right level of business protection.