Let's learn from those who have had experience with these disasters, and learn about what they did to keep their data center safe. Although one option is to shift operations to a recovery site, this column will focus on riding out the storm in the primary data center.
Understanding the risks of a hurricane disaster
From an overall business perspective, the first thing to consider in hurricane disaster recovery planning is the applications that run in a data center. It seems incomprehensible, but many data center managers simply do not know what applications they have in their data center. Applications may pile up over the years much faster than documentation, so management tends to find out about some applications when they are unavailable. Therefore, management must focus on the major applications and encourage business users to be prepared for what may be a short- or long-term outage.
If they have not been updated already, hurricane season is the time to review data center risk analyses. It is clear that the potential cause of damage is a big storm, but the actual destruction and damage may be caused by winds, falling water and rising water as the storm passes. In terms of data center operations, there need to be plans for destruction, damage, inaccessibility and utility interruptions. At a more micro level, data center managers should consider boarding up glass walls, ensuring that generator fuel tanks are full and that heavy weather supplies (especially polyethylene sheeting to cover equipment) are on hand. Dual power feeds and network demarks provide key redundancies; if they are not implemented now, it is too late to get them, but they should be seriously considered for the future.
Data center personnel preparations
Before and during hurricane season, you should conduct drills in your data center. Data centers should have a list of critical processes that follow the clock (for pre- or post-disaster activities), including exchange of backup tapes, run scheduling, remote console monitoring, preventive maintenance, shift hand-offs, print distribution, etc. The people who staff data centers need to be reminded of what they need to do if circumstances turn rapidly for the worse. And they need to practice those emergency procedures so that they can overcome incipient panic if they ever need to execute them for real.
Moreover, it is likely that if a hurricane strikes, many of the data center staff will not be there. Some will be dealing with the emergencies in their own homes, while others may be cut off from their workplaces. If a hurricane is predicted, many people will simply not show up to work. For those employees that do show up, data centers should to be prepared to offer living facilities (food, cots, showers, etc.). Also make sure that your data center is prepared to carry out functions with reduced workforces. In general, data centers are well-built facilities and likely to be safer than homes or hotels, but people understandably do not want to be away from their families during an emergency.
Senior-level management needs to conduct hurricane disaster recovery planning as well. They need to have a crisis management program to deal with data center outages well before a hurricane strikes because it takes too long to develop a crisis management plan on the spot. If a crisis management plan doesn't exist, management should make an effort to assign a senior-level team to make strategic decisions, keep customers informed and communicate throughout the organization.
Dealing with communications in a hurricane
Communication during and after a disaster can be difficult, especially during a hurricane. For that reason, it is important for data center management and staff to know how to reach one another through all the normal channels, including home phones, cell phones and personal email accounts. However, these forms of communication could be interrupted, so it might be useful to know spouses' contact information or to have out-of-region locations where people can leave messages for one another. These forms of communication can be confirmed in advance of predictable bad weather. Often, when one channel is blocked another one can be still be used; it pays to know all of them in advance.
Third parties are valuable aids in communicating during a major disaster like a hurricane. Local media outlets can be a good option to get word out to employees. Keep in mind that they are more likely to be cooperative if they are contacted in advance and know that they can count on data center staff as ad hoc weather reporters. These employees may give updates on the winds and the rain but should not comment on the effect a storm is having on continued IT operations.
If data center management gets advanced warning of a hurricane, they should notify equipment vendors at once so that they can be prepared for emergency shipment of necessary gear. Most vendors are more than willing to help, but they can ship equipment faster if they are aware in advance that shipments might be needed.
Finally, data centers in hurricane-prone areas, especially those located in isolated areas such as industrial parks, should have a satellite phone available to call for emergency life and safety assistance. The phone (or more than one if needed) should be charged and tested as a storm approaches.
There are no good disasters. But there is enough experience with hurricane disaster recovery planning to make them more manageable. They are foreseeable disasters with a long period of onset and a relatively short duration, although the aftermath may be extensive. Careful thought and preparation can do a great deal to aid in continued IT operations, even if Big Bertha (or Bradley) rolls in.
About this author: Steven Ross is an Executive Principal of Risk Masters Inc. and holds certification as a Master Business Continuity Professional (MBCP). He is a specialist in Business Continuity Management, Crisis Management and IT Disaster Recovery Planning. He is editor of the multi-volume series, "e-Commerce Security," and author of several of the books in the series, including "e-Commerce Security: Business Continuity Planning."
This was first published in August 2010