By definition, a hot site is an active IT operation that essentially copies, mirrors or replicates a primary IT operation at another site. It is ready to take over and perform primary processing if the primary site should fail or become inaccessible for any reason. There are several companies who are eager to help you set up your own hot site, or who will provide a hot site for you under some kind of contractual arrangement. Whatever...
terms and conditions govern your hot site, here's a list of key questions that must be addressed to make sure you're getting what you pay for and that you can meet your requirements for resuming normal IT activity and processing at the hot site when your primary site goes down.
1. How long will it take to switch over from the primary to the hot site?
Usually, the faster the required switchover time, the more it costs to set up, operate and maintain a hot site. Even a four-hour time window from primary failure to hot site operation can result in substantial savings vs. an "instant failover" solution. Be sure to assess risks and costs carefully, and to balance hot site costs against risk assessments.
2. How long will it take for the hot site to become fully operational? How is the cutover process managed?
Even if the hardware is ready in minutes, the IT staff must still move from the primary site to the hot site, or backup staff must be called in to take over operations. The process of a complete switchover involves more than just backed-up systems, current databases and application stores and other technology elements. Be sure to plan and practice for a quick, effective switchover.
3. What kinds of data transfer mechanisms provide current data to the hot site?
If backup tapes must be delivered, or backup images must be downloaded from an online vault, operator activity will be required and it will take some time to complete. Be sure to factor this into your switchover planning and practice, and to streamline these processes as much as possible.
4. How closely does the hot site resemble the primary site?
What adjustments will this require? IT staff accustomed to particular platforms, hardware and tools will have to adjust to any differences between the primary and hot sites. Planning and practice is especially important when virtualized solutions replace discrete servers (as is often the case at hot sites) and when tools, scripts or system interactions differ between the primary and hot sites.
5. How is staff trained to deal with disaster and to make the transition from primary to hot site?
There's no substitute for regularly scheduled drills to keep IT staff's business continuity or disaster recovery (DR) skills sharp and fresh. It's also essential to maintain ongoing training for new staff, to adjust to changes in the primary or hot site environments, and to manage the transition from one site to another. Yearly drills are mandatory; semi-annual or quarterly drills (which may involve a subset of the staff, and be scheduled on long weekends) will probably be more workable, and ensure better results in a real emergency.
6. What kinds of opportunities for practice "fire drills" are in place? How often do they occur? What aspects of the IT infrastructure get exercised?
The better and more frequent the practice in carrying out primary to hot site switchover, the more smoothly things will go in a real emergency.
7. How secure is the hot site? How available? How accessible? Is it protected against natural disaster and other threats?
The hot site must be sufficiently far enough away from the primary site so that it's relatively unlikely that both would be affected by a single disaster, yet close enough to enable smooth speedy staff movement from one to the other (or alternative staffing arranged). A hot site must also be readily available within your required cutover time interval, and sufficiently provisioned with equipment and bandwidth to accommodate current peak loads and activity levels. The site must also be physically secure, able to authenticate and monitor staff activity and sufficiently hardened to withstand disaster and (possibly) hostile situations.
8. How will compliance be maintained at the hot site? Even when disaster occurs, necessary activity monitoring and logging must continue, and privacy or confidentiality safeguards maintained. Regular drills should audit compliance activities, and take necessary steps to maintain compliance.
9. Can a hot site serve multiple roles?
Because hot sites can entail a significant and ongoing expense, more and more companies seek to obtain additional ROI from hot sites. Aside from providing backup operational capacity, many outfits seek to obtain added value from hot sites that function as extended data stores, backup repositories, provide overflow or peak processing services. Be sure to investigate such options and make sure they won't interfere with DR when it's needed.
10. How do changes to the primary site get reflected at the hot site?
The worst switchovers uncover aspects of a primary site's services, applications or data that are unaccounted for and unrecoverable. Another benefit of regular practice drills is that they will invariably highlight such oversights, and help to plan for their remediation. A careful, consistent change management scheme needs to be implemented, however, to keep primary and hot sites as closely synchronized as possible.
Like the old "How do you get to Carnegie Hall" joke, the secret to swift and effective DR remains "practice, practice, practice"— and then "analyze, plan and practice some more". By carefully considering these questions, companies can look for ways to ensure their hot sites deliver on their promised capabilities and seek for other ways to use them to add value.
Ed Tittel is a long-time freelance writer and trainer who specializes in topics related to networking, information security and markup languages. He writes for numerous TechTarget.com Web sites, and just finished the 4th edition of The CISSP Study Guide for Sybex/Wiley (ISBN-13: 978-0470276886).