Problem solve Get help with specific problems with your technologies, process and projects.

From the trenches: DR planning war stories

DR planning war stories

In homage to the comic strip, The Far Side, Steven Spielberg included a sequence in his 1993 adventure film, Jurassic Park, in which the heroes are being chased in an open jeep by a hungry Tyrannosaurus Rex. At one point, the driver glances into the passenger side mirror to see the dinosaur bearing down on the vehicle, its frightening image framed by the familiar words "Caution: Objects in mirror are closer than they appear." The scene has a tendency to evoke a nervous laugh even from those who have seen the movie several times.

It is the perfect metaphor to describe the typical corporate response to the threat of business interruption due to natural or man-made disasters. In the immediate aftermath of a disaster, there was a predictable up-tick in interest in disaster preparedness among both business and technical groups within modern organizations. Yet, as with other highly publicized disaster events, the heightened post Sept. 11 concern with business continuity planning and disaster preparedness has tended to fade as the disaster itself moved into the rear view mirror of corporate consciousness.

More and more IT managers in private and public organizations are reporting that their DR planning efforts have fallen on hard times as immediate budgetary concerns have moved preparedness efforts to the back burner. In some cases, management has lost interest, while in others the challenges associated with implementing effective strategies have proven daunting.

In the federal government setting, DR planning "war stories" ranged from the ridiculous to the infuriating. According to a field service engineer at Legato Systems, one post Sept. 11 implementation of a tape backup solution for a US intelligence agency produced an interesting conundrum.

"Legato's backup software, NetWorker, was part of the solution provided to the agency by an integrator that included a tape library and several other components. Soon after the installation, we were contacted by the customer who was very upset that he couldn't retrieve any data from his backup. For some reason, the software company always gets blamed whenever the system breaks down and that was the case here," said the engineer.

In order to troubleshoot the problem at the customer facility, one of Legato's field engineers needed to undergo extensive background checks pursuant to receiving a security clearance. "When our guy was cleared, he arrived at the site and was escorted around at gunpoint as he tried to find the cause of the problem. It turned out to be pretty simple: The client had loaded their entire tape library subsystem with cleaning tapes. Apparently, when a box of cleaning tapes had been received, someone had put "Top Secret" labels on each cartridge over the label that explicitly stated that the tapes were not to be used for recording data, but just for cleaning drive heads. For a while, the agency probably had the lowest media cost library in the history of well as the cleanest read write heads in the industry."

At another agency, an IT manager reported a different set of problems around data protection and backup. His agency, which provides a critical financial management role for the civilian departments of the federal government, relied on several hundred Windows servers to fulfill its critical tasks. Said the manager, "None of these servers have ever been successfully backed up," creating enormous exposure and risk for a vital aspect of US government operations. "Had the Sept. 11 terrorists hit our building instead of the Pentagon, it would have brought many government operations to a complete halt," he said.

The manager went on to note that two of his supervisors had brought the issue to the attention of the agency head and quickly found themselves without jobs, ostensibly for pointing out the absence of a backup strategy and the vulnerability it created. That experience convinced the IT manager about the truth in the assertion that the best disaster recovery plan is an up-to-date resume.

A third government-related story obtained second hand from a systems integrator referred to a recent acquisition of network-attached storage technology by a large agency seeking a data protection strategy. Said the integrator, "The NAS vendor had sold the agency on the idea of snapshot mirroring –- a technology for taking periodic images of file 'pointers' in a data storage environment so that, if the application fails, you can roll back to a previous version of the storage environment that existed before the problem arose. For some reason, the government client had gotten the idea that snapshots eliminated the need for data backup. Restoring from a snapped environment assumes that the data still exists somewhere."

Said the integrator, "It should have been a warning sign to the agency that several hundred NAS boxes were being 'backed up' to a single NAS appliance of approximately the same capacity as all the rest. All that the backup NAS was storing was pointers, not the data itself. So, it wasn't technically speaking a data backup solution." Subsequently, the decision-makers at the agency, who had prided themselves on having discovered an implemented such a "cost-effective" backup solution, found that they needed to reopen the issue of data protection and make another (and significantly more expensive) investment in tape backup.

The above stories from the trenches illustrate some of the typical problems that have cropped up in disaster recovery planning efforts post-9/11. Others would have to include flagging interest among business managers in spending money in the current economic climate on a capability that in the best of possible circumstances would never need to be used.

In the final analysis, disaster isn't something that we prefer to think about, let alone something we prefer to spend increasingly limited corporate resources to forestall or mitigate. All things considered, it is preferable to build recovery into our environment from the outset rather than submitting a discretionary bill to management once our systems and storage are already up and running and vulnerable.

The trick to disaster recovery planning is tenacity -– a willingness to keep plodding through the iterative cycle of testing, improvement, and retesting of strategies until they work. Just keep in mind that like the T-Rex in Jurassic Park, threats are always closer than they appear.

About the author: Jon William Toigo has authored hundreds of articles on storage and technology and authors the monthly "Toigo's Take on Storage" expert column. He is also a frequent site contributor on the subjects of storage management, disaster recovery and enterprise storage. Toigo has authored a number of storage books, including Disaster recovery planning: Preparing for the unthinkable, 3/e.

Dig Deeper on Disaster recovery planning - management

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.