Here's my list of items that companies often discover missing in their disaster recovery plans, policies, procedures and documentation:
- Authentication and validation tools: All too often, the IT staff discovers that copies of crucial SSL certificates and important account passwords or physical access devices are missing when recovery is already underway.
The solution: arrange for secure offsite storage of physical devices and for secure online storage of passwords and certificates with a third party. Practice their retrieval and use as part of your DR/BC drills, exercises and mockups.
- Personnel contacts, info and methods: In a surprising number of cases, staff members discover that they can't reach the contacts identified in the disaster recovery or business continuity plan and plans too often fail to list a sufficient number of staff members to guarantee that a valid contact is available. Likewise, personnel notifications too often rely on somebody to manually initiate such contact by phone, email or other means.
The solution: make sure a sufficient number of contacts is included to protect recovery point objectives (RPOs) and recovery time objectives (RTOs). Email and pager notification of key staff members should be arranged through a secure email account offsite. Some well-known providers include Yahoo, Gmail or MSN, where you email passwords encrypted using a tool like this password applet [http://angel.net/~nic/passwd.sha1.1a.html], and make the account and password available to all responsible parties on the recovery plan staff list.
- Geographical risks and factors: Companies operate in earthquake zones, floodplains, fire hazard areas and occasionally even in war zones without adequately planning for natural or manmade disasters. Check your situation carefully, and model the most likely disasters as accurately as you can when conducting practice drills. Make sure offsite or distant alternatives are identified in personnel information in case local staff is unavailable.
- Recovery of individual computing: During recovery, individuals and corporate IT assets such as servers and storage farms (SANs or NAS servers), need to get back to work. Make sure disaster practice addresses issues involved in providing desktop or notebook access to key staff members during recovery, and to important staff members during the return to operational status.
- Procure sufficient backup power and facilities: Many companies discover that they can't draw on adequate power or facilities when they go into recovery mode. Practice sessions will quickly identify and help suggest remedies to such problems, but they can stymie recovery or continuity as surely as the lack of other important resources. Lack of sufficient power and facilities will show up during practice drills when and as drill teams try and fail to bring systems up because of power- or facilities-related issues or problems.
- Identify priority order for resource recovery: If servers need access to a storage server or farm before they can deliver access to key services or information, those resources must be ready before or as the servers come online. In general, most network resources will be unavailable until directory services are up, so they should be brought up first. Identify key dependencies and take them into account when documenting and describing recovery processes and procedures.
- Provide adequate documentation and instructions for recovery: Beyond addressing essential dependency issues covered in the preceding item, many companies discover during recovery that some aspects of their processes and procedures are missing, insufficiently detailed or lacking important information. Creating "The Book" and going by that book during practice drills helps highlight oversights and omissions and see them addressed before genuine disaster or business interruption strikes.
- Exercise DR/BC plans regularly and rigorously: At least yearly, companies must work their way through DR/BC plans completely and thoroughly and dispassionately record all issues, oversights, omissions and errors for quick follow-up remediation. There is no substitute for practice and thorough testing in this arena.
- Keep your DR/BC plans current and corrected: It's essential to put processes in place that require staff to report regularly on plan status, and enact change management to keep plans in synch with organizational and technical realities on the ground. It's also important to perform regular audits to check how well plans and reality match.
- Regular attention and involvement: Executives, IT staff, key department heads and other staffers must remain aware and tuned into disaster recovery and business continuity needs, priorities and information. Some companies go so far as to make recovery drill participation mandatory, and a checkbox item for annual or periodic performance and salary reviews (tying participation to raises seems to be a powerful motivator). Ultimately, that's the only way to make sure that DR/BC plans completely address the return to business as usual even when disaster strikes.
Ed Tittel is a long-time freelance writer and trainer who specializes in topics related to networking, information security, and markup languages. He writes for numerous TechTarget.com Web sites, and recently finished the 4th edition of The CISSP Study Guide for Sybex/Wiley (ISBN-13: 978-0470276886).
Do you have comments on this tip? Let us know. Please let others know how useful this tip was via the rating scale below.
Do you know a helpful storage tip, timesaver or workaround? Email the editors to talk about writing for SearchDisasterRecovery.com.
This was first published in January 2009