When Hurricane Katrina devastated New Orleans in 2005, the IT staff at Tidewater Marine had to pack up all the company's servers with DAS and truck them to a secondary data center location in Houston. "We were back online within 54 hours in Houston, just in time for [Hurricane] Rita to hit," said IT director John Chaffe. "We had to pack up the servers again, box them up and fly them out to Dallas."
As recovery from Katrina began, the company's CEO insisted on a new networked storage system with off-site replication. Tidewater Marine, which owns and operates offshore oil support vehicles, chose Compellent's SAN for its multiprotocol support and boot-from-SAN features that allow Tidewater to keep diskless servers at its warm standby site in Dallas.
As with thin provisioning, where space is not allocated for data until it's actually written, Compellent's updated software sends only changed packets over the WAN,/ rather than replicating volume-to-volume on an identical array at the other end of the wire. This means it doesn't send empty blocks or require the preallocation of space on the other end of the wire for data.
Tidewater Marine's most mission-critical system is the GroupWise email server, which the company uses to communicate with its 300 to 400 vessels at sea and with its customers. "GroupWise performs a lot of garbage collection, and reindexes, rebuilds and deletes old files, which just thrashes disk and could mean we're sending the same blocks over the wire over and over again," Chaffe said. "You might have 100 GB to 160 GB of changes in two hours, and then you realize those are all beating on the same 10 GB space." Compellent's data deduplication prior to replicating over the wire keeps bandwidth manageable.
Fearing that New Orleans would lose power or worse as Gustav bore down on the Louisiana coast, Tidewater Marine failed its operations over to the Dallas data center. The process went without major problems, although with frequent "live tests" there's always something to improve for next time, Chaffe said. For example, the company had placed a single-controller SATA-based system at the secondary data center to save money. "All that housecleaning GroupWise does just brought that system to its knees," Chaffe said. "We're going dual in the future."
The company also encountered a glitch with its server and host bus adapter (HBA) compatibility. Tidewater Marine runs Windows 2003 R2 servers at its production data center with newer machines running R2 Enterprise Edition at the secondary site. That caused a problem because 2003 R2 operating systems don't see a difference between QLogic's 2342 and 2340 HBAs, but Enterprise Edition systems do. Chaffe found himself following his field engineer from New Orleans to Dallas the morning after the initial failover was performed with the correct HBAs.
As for Compellent, "I'd like to see them put more functionality into their Web-based GUI, rather than Enterprise Manager, which is a thick Java client," Chaffe said.
Moss & Associates: Frequent disaster recovery tests on the coast of Florida
Another Compellent customer, construction management company Moss & Associates, has yet to find itself in the direct path of a hurricane. But its Ft. Lauderdale, Fla., headquarters is on Florida's east coast, and director of IT Bill Snow said he has to be prepared for the worst.
Moss' data has also jumped from between 7 TB and 8 TB to between 12 TB and 14 TB over the last year or so, and Snow said some departments "will consume as much space as I give them." That makes Compellent's thin provisioning on both primary and secondary SANs a key feature. "If we have 4 GB allocated, we'll tell the end user they have two, then use the 'high water mark' feature with Compellent so we can give them more space when they get close to the limit we gave them," he said.
With space at a premium at the company's collocated disaster recovery site, not having to overprovision storage is important. "We've also added larger capacity drives because we're still replicating a lot of data," Snow said. Moss has a 100 Mbps pipe between its primary and secondary data centers, and it hasn't turned on the WAN optimization features Compellent offers because it has enough bandwidth for the replication, so far.
Since owning the Compellent box, Moss has only gone into the preliminary steps of its disaster recovery plan once, during a warning about Hurricane Fay. But the company tests frequently, and Snow said users don't notice whether they're working from the primary or secondary sites.
Like Chaffe, Snow said he had some problems with Enterprise Manager. "They've added a lot of features recently which make it easier for new users, but the advanced features are a little harder to get to," he said.
OfficeWare: Data reduction enables bandwidth-constrained replication
Cincinnati-based OfficeWare doesn't have to worry about hurricanes and only occasionally has to heed tornado warnings. But off-site replication for the company wouldn't be possible without Compellent's thin replication, said chief technology officer Chris Resch.
"We use our secondary site for testing and staging before putting things into production, as well as disaster recovery," Resch said. "We replicate volumes hourly, daily or weekly, depending on the data." The company has a 300 Mbps link, but it's filled with lots of traffic, including remote users accessing the company's storage systems and data sent over the wire for nightly backups to tape at the primary data center.
Compellent's data deduplication has cut 10 TB at the primary data center down to 7 TB at the secondary data center, according to Resch – and offered similar savings in bandwidth over the wire.
While the primary data center is in a secure colocation facility, the disaster recovery system is in one of the company's less physically secure offices. "Encryption at the source and target for this data would be nice," Resch said.