Although Health Network Laboratories doesn't quite face disasters of biblical proportions, the company's CIO said...
he adopted a "Noah's Ark" approach to disaster recovery.
Health Network Labs sells laboratory diagnostic testing services to hospitals, doctors and other health systems. Results from its labs drive most diagnostic decisions, so quick and reliable access to information is critical to its business. Following rapid growth in labs and other facilities, CIO Harvey Guindi, in 2015, decided to invest heavily in storage, servers and connectivity for business continuity.
"I jokingly call it a Noah's Ark implementation because we said, 'Give us two of everything,'" Guindi said. "We literally have an exact mirror image of solutions at our primary data center and our DR site. That gives us true, end-to-end, seamless business continuity. It's not a project you get to do very often."
Two of everything for Health Network Labs included pairs of EMC XtremIO all-flash and VNX arrays, EMC Vplex for active-active site recovery, and Cisco Nexus switches, blade servers and C240 rack servers. One of each of those products resides in data centers 15 miles apart in eastern Pennsylvania.
"We can run anything from our main data center or DR site," Guindi said. "For many things, we're load balancing between the sites."
Getting the new DR system up and running
Guindi said the DR revamp was done less to guard against natural disasters than to guarantee continuity during more mundane business interruptions. He said since implementing all his new DR equipment in 2015, the most major problem occurred when a circuit went down, severing the connection to a cloud provider for one of Health Network Labs' platforms.
"We didn't know the circuit was down. The failover process took over, rerouted our VPNs, and everything was running like nothing happened," he said.
Even a January blizzard that dumped three feet of snow on the region failed to disrupt the IT infrastructure.
"Thankfully, we haven't had any disasters [since setting up the DR system]," Guindi said. "The blizzard was more of a stress of the operational side of things than the infrastructure side."
Guindi said he has tested his system's failover and failback and is confident everything will work if there is a disaster.
"We have forced failover, and that's allowed us to do upgrades and integration changes seamlessly," he said. "We push users from one UCS in the main data center to the DR site, and they don't even know it. We can take servers down, do patch management, upgrade them, add nodes to the cluster and change storage requirements. We've leveraged this highly available, business continuity-focused infrastructure more for improved management without downtime than for actual disaster scenarios."
An all-star VDI setup
Health Network Labs' virtual desktop infrastructure (VDI) helps enable the non-downtime process. The 1,000-employee Health Network Labs has more than 700 VDI virtual desktops. They are all persistent, which means users keep their own desktop wherever they go. Health Network Labs uses Vplex's active-active capability to move virtual machines from one site to another without performance impact. If the IT team has to make upgrades to any data center hardware or software, they can move VMs to the secondary site and keep running.
"Because of the infrastructure we have, we're able to re-point people's desktops and system access at any point between the DR site and primary data center," Guindi said.
VDI is crucial for keeping the entire business running smoothly. Health Network Labs is a 24/7 operation, and each shift change can lead to a boot storm.
The high performance requirement led Guindi to acquire EMC's all-flash array for VDI storage. Guindi said user login time is around 10 seconds. He said XtremIO's dedupe enables it to hold data for all its virtual desktops in about 50 gigabytes of storage.
"We didn't want to give up performance for the sake of reliability and business continuity," Guindi said. "We really wanted to have it all. We wanted to make sure each user had a dedicated virtual desktop experience. Most of our users on physical desktops who have experienced the virtual desktop actually ask us to convert them to virtuals."
Health Network Labs replaced Dell servers and Dell Compellent storage with its current setup. Guindi said he first looked at Hewlett Packard Enterprise Moonshot servers and HPE 3PAR StoreServ storage. He liked that Moonshot's microserver architecture gives each user a separate CPU and hard drive, and thought that would be a good fit for his virtual desktops. However, HPE lacked the availability features of Vplex.
"We started down that [Moonshot] path but ran into roadblocks on replication, recovery and continuity," Guindi said. "We wanted to have seamless failover to a DR site, and [HPE] really didn't have anything like Vplex and didn't have anything built into the Moonshot platform to do that."
Keep your risk assessment process simple
Explore alternatives to formal BC/DR plans
Consider replication for your data protection strategy