A disaster recovery test plan is a critical part of any DR strategy. Without a solid DR plan and regular testing,...
organizations cannot be completely confident in their ability to successfully recover and resume critical systems, networks and other IT resources following a disruptive incident.
Here are 15 tips that ensure your organization receives the best possible results from its DR tests.
- Secure management approval and funding for the test. It helps to have a detailed test plan. Prepare a draft test plan that can be submitted to management along with your funding request. If a disaster recovery test plan request must be processed using your organization's change management process, follow those steps carefully.
- Provide detailed information about the test, including the nature of the test, its goals and objectives, success factors, test procedures and post-test analyses. Failure to include these and other activities in a disaster recovery test plan may result in its rejection by management and a loss of funding.
- Identify the test team, secure its availability on the planned test date and note additional subject matter experts, such as database administrators and network engineers. Without the right DR test team in place, it may be difficult to complete the test and obtain the best possible results.
- Determine what will be tested and the expected outcomes. Are recovery and resumption processes for systems and networks being tested? Will a data backup and recovery process be tested? Will you test procedures to notify employees using an automated notification system? Will the notification system itself be tested?
- Document the disaster recovery test plan carefully, especially test scripts and specific scenarios that may be introduced to change the parameters of the test. An initial DR test script gets you started, but be prepared to edit it and possibly alter it significantly by the time the test has concluded.
The percent of companies that test their disaster recovery plans at least twice a year.
Source: TechTarget Survey
- Confirm test scripts are correct. Scripts are the playbooks used to execute tests, and their accuracy is essential. Even transposing two characters in a line of code can cause a system test to fail. If possible, have a subject matter expert review the script before the test to identify any possible changes or edits.
- Incorporate all technology elements needed for the test into the plan. These include network, hardware, application, database, utilities and any other special arrangements that are unique to the system(s) being recovered or processes being validated. The seemingly insignificant item you overlook may make all the difference in a successful DR test. If one or more components are not ready, cancel and reschedule the test.
- Verify that the test environment -- such as R&D -- is ready for the test, so you won't disrupt production systems. Scheduling of this environment is very important, so be sure to factor this into your request for test time.
- Ensure your test does not conflict with other scheduled tests or activities. In a medium- to large-sized IT organization, there will likely be many scheduled activities in place, such as new system acceptance testing, network upgrades and software upgrades. Your test will need to fit in between other scheduled activities to avoid disruptions to other work.
- DR tests often can take several hours, so schedule them as far in advance as possible. Send out notifications -- and reminders closer to the test date -- to other IT leads and managers advising them of your planned test.
- Schedule a dry run of the test before going live. You may be able to spot problems, fix them in advance of the test and increase your chances of a successful test.
- Be prepared to halt the test and reschedule it if needed. If it is clear, after the initial test steps have been performed, that things are not going as planned, stop the DR test and review what happened up to the point where the test stopped. If it's possible to bypass the element that failed and continue with the test, proceed with the original script.
- Have a scribe take notes. Add a timekeeper to record start and end times. Both players are important as their notes will help to prepare an after-action report on the test.
- Complete an after-action report that documents what happened during the test, what worked and what didn't, and lessons learned. Record times when steps are completed and then document changes to test scripts developed on the fly. Note that auditors may wish to review how the test was conducted, actions of the specific test participants, how specific issues were handled, the infrastructure elements that didn't work and results achieved.
- Use results from the test to update DR plans, business continuity plans and other documents. Disaster recovery test plan documents should accurately reflect what the organization learned from the tests.
Improve resiliency by adding new technology to your DR test plan
Use monitoring tools for smooth DR plan testing
Tips on testing a cloud DR service
After-action reviews can improve future DR tests