The devastating tornado that blasted parts of Alabama in 2011 prompted the executives of Morris, Ala.-based Reno Refractories to revamp their disaster recovery plan. The building materials manufacturer upgraded its backup and networking infrastructure, and added a disaster recovery cloud to its high availability strategy.
Wayne Bailey, Reno Refractories’ network administrator, said the company had no real
Bailey treated the April 2011 tornadoes that hit about 50 miles away as a warning. He said he toured the Tuscaloosa area after the storm with his son, a Tuscaloosa county deputy sheriff, and saw “nothing left but bare dirt and concrete” where large buildings stood before. Upon returning home, he saw another sign of the storm’s power when he found a check that was cashed from a Tuscaloosa bank, a flyer from a Tuscaloosa insurance company and a picture from a Tuscaloosa-area yearbook in his yard.
“When I saw that, I sat down with the president of our company and said, ‘What we’re doing is not enough,’” Bailey said. “What hit me from what I saw in Tuscaloosa is, one day people went to work and everything was normal. But they didn’t just have a disaster; their entire business identity was sucked up into the tornado and gone. These businesses didn’t know who owed them money, who they owed money to, what their assets were, what their liabilities were. We have all kinds of insurance for replacing buildings, but our business identity is within the data in our systems.”
With fewer than 100 employees and one IT person, Reno Refractories’ previous DR plan was backing up to tape and moving the tapes to an off-site location. “But how long would it take for us to recover, and what could we recover?” Bailey wondered. “We could recover back to the last day before the disaster as far as what was in our ERP system. That was our inventories and financial system. A lot of other data, such as CAD file drawings, would be history. That prompted me to start searching for something we could do here within our financial capabilities.”
Among the options he considered was to upgrade Reno’s EMC Clariion AX4 SAN, buy a second SAN for a DR site, and replicate between them. Besides the expense, it would require a lot of work to get his network ready to replicate between sites, Bailey said.
“That would be a nightmare of networking and address IP schemas,” Bailey said. “You’d have to make IP changes, create another domain, do subnetting, and the list went on and on.”
Bailey went to a trade show last fall and attended sessions in its DR track. He looked at a combination of Veeam backup software for virtual machines and an ExaGrid data deduplication appliance, but Veeam did not support the Citrix XenServer hypervisor that Reno uses. Data protection offerings from large storage vendors were out of his price range.
He was invited to a luncheon that Quorum hosted in Birmingham around the same time. The vendor’s presentation convinced Bailey the onQ appliance would be simple to implement and “the dollars and cents fell into place.” He finalized the deal in December and went into production in January. Bailey said he installed the appliance himself. The onQ appliance replicates data to the Quorum-hosted cloud in Fremont, Calif.
Reno Refractories uses Quorum as its HA solution for local files and application servers hosting its ERP system, Microsoft SQL and Exchange and Citrix XenApp and XenDesktop.
“We run backups every two hours,” Bailey said. “Our primary application server houses our ERP software and Microsoft SQL servers and databases, and they get backed up every two hours. We also back up our Exchange server every two hours, virtual domain controllers every six hours and our XenApp server every four hours. Two other servers related to VDI [virtual desktop infrastructure] get backed up twice a day because there’s not a lot of change on VDI machines, and the servers aren’t critical for running virtual desktops.
The onQ device can also provide relief for when servers go down or users accidentally delete files. “If somebody says, ‘I accidentally deleted this and I need it back,’ I tell him, ‘I can’t guarantee the last document you made changes to, but I can probably get the document prior to that,’” Bailey said.
Reno’s onQ appliance consists of two backup nodes and four hot DR nodes that reside in the DR cloud. Bailey said the onQ device lets him test restores, which he takes advantage of nightly for several applications.
“I set up a node for Exchange, for instance, and it backs up every two hours,” he said. “I pick a time at night when nobody is here, and the device automatically fires up my Exchange server, brings it to full operational mode and lets it run a couple of minutes before shutting down. Every day at nine a.m., I get an email from the device about how many backups took place over the last 24 hours, and if the start-up tests of my virtual machines passed or failed.”
He can also use onQ’s web interface to start a recovery node in test mode.
Bailey also upgraded Reno’s networking infrastructure after the 2011 tornadoes. He kept his PRI-T1 lines for voice, but upgraded his T1 data line to a metro Ethernet connection. He now has 10 Mb Ethernet that is burstable to 100 Mb.
On the facilities side, Reno Refractories’ employee break room serves as a tornado shelter that can hold all of the company’s employees. The tornado shelter was included when the factory was constructed in 2001.