The first half of 2010 has been one for the record books in many parts of the United States. Tornadoes and floods...
have devastated larges areas, causing deaths, property damage, and endangering IT operations and the security of data. For instance, in early May, the U.S. Navy Personnel Command offices in Millington, Tenn., were covered in several feet of water, putting IT operations out of business and stopping promotions and personnel transfers across the Navy for days.
On a smaller scale, a countless number of organizations across the Northeast had to deal with unexpected flooding and flood disaster management this spring. Richard Booth, of the North Kingston, Rhode Island school department, was dealing with widespread flooding across his whole community. Booth is in charge of IT operations for the community's nine schools and manages 1.6 TB of "mostly mission-critical data." According to Booth, one of the buildings where he manages operations had a significant roof leak two years ago. That incident didn't affect his operations or cause any loss of data, but it was a wakeup call for his disaster recovery (DR) strategy. So, this spring, when the water began to rise across town, shutting down roads, flooding basements, and cutting off power, Booth made sure he was prepared for this type of disaster. He equipped his community with a pair of Exagrid disk backup appliances, an EX5000 and an EX3000. Booth installed the units in two locations, at opposite ends of the town. The investment made operations more secure and also eliminated the need for a manual data backup each weekend.
"The Exagrid systems are constantly streaming data back and forth, which means I always have two copies of all my data -- if I lost one building, I could still recover my data," said Booth. To add to his disaster recovery strategy and preparation, Booth invested in raised flooring to protect his systems from relatively minor flooding and also installed a battery backup system so that in the event of a power failure he can ensure an orderly system shutdown. And for added data protection, Booth said he still generates tape a few times a month to store off site as extra insurance. It all adds up to "peace of mind," he said.
Data backup plans save the day
Suffering from the same set of storms on Long Island, NY, Andrew Hult, a local CPA who runs a successful accounting practice out of his Hempstead office, faced the scenario of a looming federal tax deadline with floods threatening to shut down power to his offices. Fortunately, he was able to get assistance from CMIT Solutions of South Nassau, a local information technology solutions provider for small- to medium-sized businesses (SMBs), that set up access to key data for the CPA firm's accountants to access office data through internet connections at their homes.
CMIT Solutions was able to do this through its remote data backup solution, downloading the needed data to the employees' home computers within hours of the flood. The disaster recovery plan the CPA firm had in place with CMIT Solutions saved time, money and frustration. Armando D'Accordo, owner of CMIT Solutions of South Nassau, said the service, CMIT Guardia, backs up data both onsite and offsite. "The service varies in cost based on the services performed and the type and amount of data that is stored offsite. This particular accountant pays $25 per month for their offsite encrypted daily backup and storage," he said. Other clients with more extensive disaster recovery plans that include offsite server images and much more data can cost $300 per month or more. "The accountant in question had 7 GB of essential data that had to be restored onto systems that were relocated to a place with power, as their regular office was without power for six days," said D'Accordo.
According to D'Accordo, technology disruptions to any business can be a traumatic blow, but it's more traumatic for a CPA firm during tax season. CMIT had to contend with the flood disaster; at one point they were without power for four days. However, CMIT was still able to function and help their clients because of their emergency generators. "Our first choice was to continue to work from our office, however, we knew that we were prepared for option B if it was needed," said D'Accordo. "Every business should have an option B in place. The plan should include offsite backup of data, a second location to work from and it must be actionable and appropriate to the size of the business."
Other businesses faced similar disaster recovery issues because of the floods this past year. In Massachusetts, Realtor Pam Sheehan, of Sheehan Realty in Taunton, and an attorney in an adjacent office both suffered extensive damage from about two feet of flood waters. However, her last-minute decision of putting their PCs on top of their desks and forwarding calls to her home kept her business running. Also, Sheehan noted that most of her data was never at risk because she makes periodic backups onto optical disks, which she stores at home. So overall, the long-term impact of the flood was fairly minimal to her company. They just had to replace the rugs and repair the sheetrock.
Disaster recovery strategies and preparation a must
Jim Lippie, vice president, Staples Network Services, an IT services company that targets the SMB market, said companies are sometimes their own worst enemies when it comes to preparing for disasters. "More often than we'd like, we see equipment placement minefields in SMB environments -- whether it's improper ventilation and lack of an adequate cooling strategy, equipment littering the floor, or water damage waiting to happen," he said. For instance, Lippie said, at one company his team made an alarming discovery -- the server room also housed the company's hot water heater. "On the spot, our engineer insisted on setting up a tape backup," which later proved crucial to recovery when a leak did develop, said Lippie.
However, warned Lippie, counting on luck or last-minute fixes is a high-risk strategy. Thus, he recommends having a third-party assessment of your data backup environment "at least every two years" to look for and fix any potential hazards. Another option -- popular for its time and cost savings -- is looking into a collocation data center to house data center equipment. Lippie said Staples Network Services itself has taken this option. From 2000 to 2007, Staples Network Services experienced at least one power outage per year ranging from 10 minutes to several hours. So in early 2007, the company moved its infrastructure to a collocated facility. Now, Lippie said he "sleeps better at night." Lippie also said that Staples Network Services is also helping a financial services firm that processes thousands of financial transaction every day and can't afford downtime, to evaluate on premise infrastructure vs. "colo". According to Lippie, determining which strategy you want to go with comes down to space, redundancy, power, cooling and associated costs.
In terms of a data backup procedure, companies that use tape storage should have a daily tape rotation. Another data backup strategy to consider is offsite data backup through online data vaulting. "Without a proven data protection plan, they're simply taking an unacceptable risk," said Lippie.
Furthermore, he noted, one-size-fits-all approaches can leave you vulnerable. He cited the example of a client located on the top floor of a multi-level office building where the building roof was getting tarred. Unbeknownst to the company, there was a small hole in the roof above the server room. "Imagine their consternation when they discovered that tar had dripped down into the server room, coated the server and rendered it completely useless," he said. Fortunately, the company had implemented a tape rotation backup and they were able to get a replacement server up and running within a day.
"While companies don't necessarily need to have servers on standby, it's important to always have an accurate and up-to-date asset inventory, including equipment specs, so you can do an overnight order if necessary to preserve business continuity," said Lippie. Again, in terms of a comprehensive disaster recovery plan, Lippie said companies should have flexibility to adjust to not-so-typical disasters, and include measures for communicating to employees and distinguishing between short- and long-term emergencies.
Less exotic and far too common is the "sin" of poor data backup practices. Lippie said he has encountered many companies that perform daily backups but without testing. Instead, they simply change tapes from day to day without ever determining if the tapes are operating correctly. "We've gone into some prospects and discovered that they haven't had a successful backup in more than six months," he said. In fact, he said, studies have shown that more than a third of companies never test their backups, and of those that do, more than three quarters have found tapes that fail to restore properly. "Needless to say, doing test restores is absolutely a must," said Lippie.
Founder and consulting analyst at the Tanjea Group, Arun Taneja, said disaster preparedness considerations are common to all sizes of business, but the responses need to be appropriate to the business size. For instance, Taneja said the "classic" advice for setting up a disaster recovery site is to place it at least 300 miles from your main operation. However, for smaller businesses, much smaller distances may still confer benefits without adding too many costs. The main point is to ensure that, as much as possible, a secondary site depends on a different set of utilities. "When there is a disaster, it is not uncommon for all of the utilities to be affected in an area," noted Taneja.
Disaster recovery planning is no longer the exclusive purview of larger companies; every company needs a DR plan, said Taneja. The good news is if you go to a virtual server environment, DR can become simpler. "You can get a single image of the system that contains everything including the applications and operating system and the data," he said. By comparison, traditional DR requires not only preserving data but setting up and replicating the hardware, software, and all patches and fixes. "That's something even big companies have trouble dealing with," said Taneja. With virtual machines (VMs), the whole image can be readily transferred to another hardware environment. "It is something small to midsize companies should really consider," he added.
Another option that is increasingly viable, said Taneja, is online data backup, or cloud backup. "The cloud is a whole new world, that provides "rudimentary data protection" built in. "Depending on the service and type you use, you might only have a backup every day so you might still lose some information but that might be acceptable for many smaller businesses," he said.
About this author: Alan Earls is a Boston-area freelance writer focused on business and technology, particularly data storage.