Figuring out how much to spend on disaster recovery (DR) is always difficult for organizations, but shrinking IT budgets make the problem even more acute. Despite these challenges, for some organizations, not even a lousy economy is an excuse to cut back on disaster recovery investments.
"Our capital budget is probably half of what it was last year, but we don't scrimp on DR spending. We'll defer a system upgrade before we defer the capital needed to maintain our DR capability," said Harry F. Lukens, CIO of Lehigh Valley Hospital (LVH) and Health Network in Allentown, Pa.
The 700-bed hospital system uses a series of "hot boxes" at a secondary data center in nearby Bethlehem. Formerly a testing and data center of IBM Corp., the facility was purchased as part of an acquisition of another hospital about 10 years ago.
The off-site data backup servers enable 14 different critical computing systems -- including those for operating rooms, medical/surgical, and labor/delivery -- to continue functioning in the event of an outage. In addition, LVH has configured individual backup servers for about 40 other major systems housed at its primary data center in Allentown.
Lukens estimated that LVH spends about $540,000 annually on disaster recovery, including capital costs of $300,000 to upgrade or replace servers. Operating expenses, including testing and a salary for a disaster recovery coordinator are about $200,000.
The disaster recovery plan is managed mostly by the hospital's IT department. The lone exception: two Tandem mainframe computers are outsourced to DR services provider SunGard of Wayne, Pa. The outsourcing "insurance" costs LVH about $3,500 per month, Lukens said.
How much disaster recovery spending is too much?
"Supporting disk-to-tape backup, in which you need to recover within a few days, is going to cost you less than having a dedicated disk-to-disk infrastructure that lets you recover in a matter of hours," said John Morency, a research director with Stamford, Conn.-based Gartner Inc.
Citing Gartner's research during the past several years, Morency said small- to midsized businesses (SMBs) devote anywhere from eight-tenths of 1% to 2.8% of their IT budgets to disaster recovery tools, training and services. The amount of disaster recovery spending is affected by an organization's infrastructure, configuration and management needs.
"The key point is to align your DR investments to ensure you have a reasonable balance between risk mitigation and affordability," Morency said.
Business impact analysis and risk in DR planning
Experts say it's difficult to forecast costs unless you identify the threats, their probability and their financial impact on your business. That process is known as a business impact analysis (BIA). It provides information that helps to pinpoint which business processes and applications are at risk and, more importantly, how quickly they need to be restored.
"The point at which you need to recover your data has a huge impact on costs and how to budget. In general, the longer you extend your recovery time, the lower your cost of recovery is going to be," said Larry Arker, a risk-management consultant with Jefferson Wells in Milwaukee.
Trying to anticipate and prevent any inconvenience at all is "exactly the wrong approach," said Richard Jones, vice president for data center strategies at Burton Group, a consulting firm in Midvale, Utah.
Several years ago, a manufacturing company in the Ohio Basin made a crucial decision: Don't worry about every application or process. Instead, Jones said the company determined that only two out of hundreds of business applications needed to be recovered within one day. Most applications were protected using inexpensive tape backup.
On the flip side, Jones said Wall Street firms stand to lose millions of dollars per broker for each minute a system is down. Therefore, they may dedicate 75% to 80% of IT budgets on disaster recovery.
"Understanding threats and probabilities gives you insight into how much money you risk losing, and how much you're going to have to spend to maintain the business," Jones said.
When preparing his annual disaster recovery budget, Lukens requires each of his server directors to provide an itemized list of hardware that will need to be replaced in the upcoming year. The "bottom-up-driven budget" ensures the wisest use of disaster recovery dollars, he said.
"You can't just say, 'Here's a bunch of money, go make it happen.' Because you may be spending too much or you may be spending too little," Lukens said.
Organizations make several overspending mistakes including trying to provide total or near-total redundancy, when lower-cost alternatives would suffice. Besides overspending, for companies using lower-cost tape media, rising energy prices are forcing them to pay higher rates for transporting backup tapes from their archival provider (such as Iron Mountain Inc. or Seagate Technology's i365) to testing sites. "It's not unusual to see rates of $5,000 to $6,000 per [archival company] truck roll. This number adds up fast when you have lots of tapes that are needed for applications and data restoration," Morency said.
Reexamine disaster recovery spending priorities
In order to prioritize spending, companies should use the slackened pace of business to decide if they are making the most of their disaster recovery budget. For example, look into whether or not you can reallocate costly storage or replication hardware to high-priority applications and shift other applications to less-costly tape backup.
It's a good time for companies to scrutinize their disaster recovery plans a little better to try and squeeze more cost savings from it," Jones said.
Scale back on DR tests
Disaster recovery tests are costly and time-consuming, so it's important for an organization to know and test only what is necessary. Email, enterprise resource planning systems, supply-chain networks, payment and payroll, intranets/extranets, and customer-facing websites are typical applications that most organizations will want to test routinely.
"If an organization doesn't do this type of analysis, then the implicit expectation is going to be that IT can recover everything, which is totally unrealistic" in most cases, Morency said.
Collocation and disaster recovery
Until the economy rebounds, few companies are willing to incur the huge capital cost associated with building new data centers. That includes postponing expansions of existing data centers to accommodate new applications. Meanwhile, companies are opting to use collocation or hosting providers such as Hewlett Packard (HP) Co., IBM Corp. and SunGard as a "tactical cost-saving step" to support backup and recovery, Morency said. Some companies are taking a blended approach, divvying up disaster recovery dollars to both expand their DR architecture while outsourcing secondary and tertiary data tiers to outside providers.
Also, as those disaster recovery contracts come up for annual renewal, an organization may be able to reduce costs by reconfiguring its environment or reducing the number of hot sites needed.
If you have a smaller IT staff that's being asked to do even more, make sure they have the appropriate level of training. Because of budget cuts, Lehigh Valley Hospital has 7% fewer IT staff this year, but disaster recovery requirements aren't slackening. "To make sure we're covering all our DR stuff, we're having to cross-train people (on different servers) now more than we ever did in the past," Lukens said.
About this author: Garry Kranz is a freelance technology writer in Richmond, Va. His work has appeared in SearchCIO.com, SearchSecurity.com, and other TechTarget news portals.