For instance, Frank Ambriz, a consultant who handles IT issues for McAllen Primary Care of Texas, said he has seen first hand what happens at organizations that don't plan for disasters. "We lost a key disk drive and at first the only option appeared to be rekeying the data," he said. Fortunately, his organization was able to recover the data with the help of Kroll OnTrack Inc. and, working informally with a value-added reseller (VAR) to improve its DR capabilities, has since implemented a basic backup process that involves regular physical movement of backup data to an off-site location.
John Morency, a research vice president at the Gartner systems security group, said the world of disaster recovery consulting is currently dominated by small, boutique operations -- there are at least 700 listed in published directories -- along with a small number of midsized and very large firms.
"What it boils down to is when you are hiring a DR consultant, ideally you want someone who has a formal methodology that they can describe and step you through as part of their proposal in terms of how they will attack each of these areas," said Morency.TABLE OF CONTENTS:
What to look for in a disaster recovery consultant
In considering how to approach a DR challenge and how to hire a consultant, Morency said it is useful to separate the business continuity and disaster recovery functions within consulting into five distinct phases:
- Conduct (or refresh) your business impact analysis (BIA) and determine where you stand.
- Define your disaster recovery strategy. This step states the requirements and defines the things you need to recover for your most mission-critical applications.
- Design and implement a recovery infrastructure. This is one of the biggest issues for companies because most companies have several applications or appliances that are in some way related to data backup or recovery.
- Define the test plan. Morency said this is probably the most critical part of an overall disaster recovery strategy because in stark contrast to the days of mainframes, with distributed and Web-based applications, there is more complexity. "Gone is the old model of working with a DR provider and testing once or twice a year. Now you need to test more frequently, perhaps focusing on certain critical areas," said Morency.
- Break down responsibilities. In this phase, determine the responsibilities of different members of the recovery, pre-test and post-test, including system and storage administration, network people and anyone involved in shaping or testing the disaster recovery plan. Too often, this doesn't happen, said Morency, and because of that, there is often a lack of specificity around who the consulting team members are. "It is the old adage that the senior team members sell the engagement and the junior people deliver it. Just having a consultant be specific about who the team members are and their backgrounds and relevant industry experience is a strong differentiator between a quality proposal and something dicey," he said.
Make sure your disaster recovery consultant is suited for your environment
Bob Laliberte, an analyst at Enterprise Strategy Group, said consultant experience is critical and must match you environment. For example, "mainframe skills are great but may not apply to a virtualized Windows environment," he said. And even within vendor environments there can be a wide range of skills. "For example, even within the EMC [Corp.] environment, a consultant might need to know RecoverPoint, SRDF [Symmetrix Remote Data Facility], or MirrorView, and if you want to go to the application side with Oracle databases and so on, and high-availability issues, it becomes obvious that there are a very wide range of technologies involved with disaster recovery," said Laliberte.
Laliberte also urges those focusing on DR planning to take a broad view of the task. "Up until Sept. 11 there were companies focused just on data and IT, and they didn't take into account the people issues involved with disaster recovery," he said. The events of Sept. 11 caused not only a terrible loss of life but revealed the complex problems companies face when recovering from a big disaster. Many of the companies affected had data backup plans with IBM and they discovered too late that whoever declares a disaster first, gets in line for services first. "Because IBM's Sterling Forest facility reached capacity quickly, many companies found they had to work out of facilities in Colorado or Maryland or some other distant location. So, even if you are only trying to think about IT, you need to think about the people involved and where they will work and, perhaps even live," said Laliberte.
Similarly, Laliberte said disaster recovery consultants should help you bring in tools to help you see what applications have dependencies. "You could be surprised to find out that an authentication program for accessing a critical app is actually running on an unprotected x86 machine," he said. Consultants range from Accenture, Hewlett-Packard (HP) Co. or SunGard at the high end and then toward the middle there are companies like Iron Mountain Inc. "or maybe a trusted reseller that, over the years, has sold you equipment and understands your needs," he said. For a smaller company, it could even be an individual.
In fact, noted Mark Finocchario, national director for recruiting at Eliassen Group, an IT recruiting firm, recruitment firms can be a source of DR expertise. "Our clients are able to obtain the same resource at a reduced cost by eliminating the middleman."
Steve Bandler, COO of Boston-based Analytics Operations Engineering, a 25-person company that provides operations consulting services, said he has learned that the most important characteristics of a disaster recovery or business continuity consultant are experience, along with analytical tools and techniques for business continuity planning and "an understanding of our type of business."
Bandler, who ended up giving his DR work to Staples Network Services, a company he already had a relationship with, said, "In addition to those characteristics, I would look for excellent references and real-world experience demonstrating successful management of a DR event," he said.
Disaster recovery consulting services aren't cheap: How to get the most for your dollar
Greg Schulz, senior analyst at the StorageIO Group, further recommends being clear about objectivity. "Some consultants are totally independent, with no vendor or hardware connection at all, but many others are tied to a vendor or at least to their value proposition," he said.
"Unfortunately, if you are a consultant selling billable hours you will tend to find a way to generate even more billable hours," he added. If you aren't sure where to begin, Schulz said an option is to start small and get some basic help scoping out what you need. Then, from there, you can determine what additional different consultants or services you might need.
It's important to determine how much time you and your staff can dedicate to the planning process, as opposed to relying on the consultant to construct the plan.
COOAnalytics Operations Engineering
Regarding disaster recovery planning budgets, Morency at Gartner said if you are just looking for a refresh of your DR plans or strategy, typically the price range for a single facility or a small campus would be in the $75,000 to $150,000 range. A large multinational might see a project price between $500,000 and $1 million, depending on the scope. If you choose a single site/campus, extending the consulting effort to include the five phases Morency defined (not including hardware/software) would probably incur charges in the $200,000 to $350,000 range. "Smaller boutique firms generally hold their hourly rates close to $200, but large organizations with wide geographic span often charge as much as $350 to $475 per hour," he noted.
"Most of the clients I talk with need to get a refresh of their business impact analysis and revisit their recovery time requirements for different production applications, define risks for different tiers -- and then put together a recovery and continuity strategy," he said. And Morency added that those tasks don't require someone with a lot of in-depth expertise around the data center or the specifics of backup and recovery. It is more for business-oriented people.
And, noted Laliberte, the business continuity consultant needs to be able to translate business requirements into balanced and cost-effective DR solutions. "It is easy for anyone to spend a ton of money and build a robust DR environment," he said.
Lastly, Analytics Operations Engineering's Bandler offered a few additional tips. "It's important to determine how much time you and your staff can dedicate to the planning process, as opposed to relying on the consultant to construct the plan," he said. "Completing more of the plan in-house saves on consulting fees, but if the plan never gets completed it may be of little use when an actual DR event occurs."
To develop the disaster recovery plan, Bandler estimated his company spent about $2,500 with Staples Network Services. Looking back on the experience, Bandler is happy he hired a consultant and said he found them to be knowledgeable and helpful. Moreover, Bandler said he would definitely make the same choice if he had to create another DR plan.
About this author: Alan Earls is a Boston-area freelance writer focused on business and technology, particularly data storage.