Virtua healthcare systems designed a disaster (DR) recovery plan to ensure there was no downtime while it moved into a new data center in New Jersey last year, and it has stuck with that DR plan
Virtua’s healthcare system includes four hospitals and 50 satellite sites in southern New Jersey. In 2009, the IT team began work to revamp its 20-year-old data center that was running short of power, cooling and space, and struggling to keep up with new technologies and data-intensive medical imaging systems. The new data center included upgraded EMC Corp. storage, expanded Cisco Fibre Channel and Ethernet switching, and a significant VMware implementation.
The tricky part was doing the data center move without downtime, because Virtua needed to keep its patient data online throughout the process.
“We had to make sure our data was available during the transition,” technology project manager Joe Collins said. “The only way we could do it was to build a new data center and migrate data.”
Virtua partnered with a colocation facility about 40 miles away, and in early 2009 installed its new storage systems there and replicated data from its original data center. It then moved servers to the colocation site in phases. Construction on the new data center began in November 2009, and when that was finished last year, the IT team began migrating data there. By then, its storage requirements had nearly doubled from when the project began.
“When we built our new data center, we took that storage that was replicated, paused replication, moved that three miles down the road to our new office, and from there we initiated replication again with minimal downtime,” Collins said. “We went from about 100 TB of active storage to where we’re now hovering with about 200 TB [active], with a maximum capacity of 300 TB.”
At no time were we out of data or out of touch. Everything was connected.
Paul Krihak, Virtua’s manager of enterprise infrastructure
Virtua’s manager of enterprise infrastructure Paul Krihak added that over the nearly two-year transition, “At no time were we out of data or out of touch. Everything was connected.”
And he intends to keep it that way. With the colocation site serving as the primary data center and the new facility as the secondary site, Krihak said Virtua can fail over between data centers in 50 milliseconds -- before any of his users would notice a difference.
Virtua has EMC CX4-480 arrays in its data centers and uses EMC MirrorView to replicate between them. It also has more than 225 VMware virtual machines (VMs) and about 300 physical servers.
“We were in the infant stages of our VMware implementation during the move,” Collins said. “We actually accelerated it, and were able to justify it by this migration of our production data center. We were concerned about damaging a lot of our legacy systems -- those fragile artifacts. We then converted them to VMware, and migrated them to the new data center.”
He said using VMs also provides quick recovery because he can take down a VM and move it between data centers in about a half hour instead of hours to move a physical server.
Along with the EMC storage (besides Clariion SANs, Virtua uses Centera for records archiving and Disk Library for backup), both data centers use Cisco MDS 9509 Fibre Channel switches. The secondary data center includes Cisco Nexus 2000, 5000 and 7000 switches with plans to add more Nexus switches to the primary data center. The data centers are connected by a Layer 10 Gbps dense wavelength division multiplexing ring with leased fiber.
Virtua’s technology transformation isn’t finished. It is putting in Fibre Channel over Ethernet (FCoE) as well as a virtual desktop infrastructure (VDI) and a new “paper light” 368-bed hospital is set to open in Vorhees, NJ, in May. The hospital will host a Health Information Exchange (HIE), which is a central repository for several health care organizations that will share information. Virtua will be one of the primary storage facilities for the HIE, and replicate data to other sites on the exchange.
Collins said he expects the FCoE implementation to be completed over the summer. It will include an upgrade to 10 Gigabit Ethernet (10 GbE), plus an installation of Cisco UCS and Nexus switches in the primary data center. He said Virtua held off on the network consolidation to FCoE until now, because “to allow for minimal downtime, we didn’t want to add too many technologies.”
Krihak said FCoE will let Virtua reduce cabling by using two 10 GbE connections instead of eight to 10 FC and GbE cables. He said the consolidated networks should also allow Virtua to provision storage faster.