Published: 10 Aug 2005
High-speed optical WANs are becoming indispensable for critical storage applications. Business continuance, remote mirroring and replication, and connecting regional data centers are all tasks that require optical WANs. With many optical options and prices, picking the best technologies and techniques to transport data is challenging.
Most storage apps are time-sensitive and require high throughput (bandwidth) and low latency with zero data loss. Effective bandwidth is a measure of how much of the available bandwidth can actually be used, taking into consideration dropped packets and retransmission due to congestion and protocol inefficiency. Some vendors, including Ciena Corp., Linthicum, MD, refer to effective bandwidth as "good put" vs. simply looking at a theoretical line speed number.
To pick a data transport to fit your needs, first consider your storage requirements:
- Distance. How far away do you need to keep a copy of your data?
- Bandwidth. How much data must be moved and in what timeframe?
- Cost. What's your budget?
- Recovery point objective. Can you afford data loss? At what point do you need to recover data from?
- Recovery time objective. How quickly do you need to recover?
- Latency. What are your applications' response time requirements?
When evaluating storage-over-distance network technology, keep the following items in mind:
- Throughput. How much bandwidth is available and how much is needed?
- Latency. What's the time delay and how does it impact the storage application?
- Packet loss. How much bandwidth is lost due to data retransmission?
- Variable bandwidth. What's the granularity of available bandwidth?
- Manageability. What are the networks' management capabilities, including diagnostics?
- Flexibility. Is the network service adaptable to changes in your storage environment and its locations?
- Resiliency. Are there diverse network paths, and does the network contain self-healing, rapid failover features?
- Budget. What are the initial upfront and recurring costs?
IP network technology gets a lot of attention, but most storage is sent over optical networks, which are well suited for applications that require low latency and congestion-free bandwidth. (Note that in the networking world, bandwidth is typically referenced in bits per second; for example, one billion bits per second or 1Gb/sec. Storage is referenced in bytes per second; for example, 100MB/sec.)
Dedicated fiber: Dark fiber refers to a dedicated long-range, single-mode, fiber-optic cable pair (send and receive) that exists between two or more points. A dark fiber cable can be purchased or rented from a carrier or service provider. Dark fiber can create a private network supporting network interfaces, including ESCON, Ethernet, Fibre Channel (FC), FICON and synchronous optical network/synchronous digital hierarchy (SONET/SDH), or can be used with wave division multiplexing (WDM). With dark fiber, it's the responsibility of the user or service provider to light and manage the fiber-optic cable, including the creation of redundant paths and traffic management. For relatively short distances (35km to 80km), ultra-long-range gigabit interface converter (GBIC) and small form-factor pluggable (SFP) optical transceivers coupled with SMF optic cable can be deployed. Dedicated, dark fiber optics can be difficult and expensive to obtain outside of a user's premise in MAN and WAN environments vs. shared-bandwidth-based options.
Wave division multiplexing: Typical fiber-optic deployments involve a fiber-optic cable that's dedicated to a specific network interface. Using WDM, a fiber-optic cable can support multiple network interfaces, each with its own private bandwidth. WDM technology sends light through an optical device using different lambdas (wavelengths) of light. Each network interface that's attached to a WDM-based wavelength functions as though it has its own virtual fiber-optic cable. Consequently, the performance of the different networks varies and each network can be dedicated to support applications with different bandwidth requirements. In addition to supporting simultaneous networks, WDM enables longer distances using long-range optical transceivers.
There are two variations of WDM and they differ in price, capacity, distance and management capabilities. The first is coarse wave division multiplexing (CWDM), which has a lower cost, lower bandwidth and spans shorter distances (80km to 100km). The other variation is dense wavelength division multiplexing (DWDM), which provides more interfaces and greater bandwidth, as well as support for longer distances at a higher price. CWDM is a good entry-level, low-cost solution for applications that have lower bandwidth and distance requirements. DWDM can support storage applications such as synchronous data mirroring and replication over distances of 300km or more; but as distances increase, so does latency.
SONET and SDH optical networks: SONET and SDH are optical-based backbone transport networks deployed by carriers and service providers for metropolitan and wide-area networking on a global basis. SONET and SDH networks support voice, data and video with guaranteed bandwidth, low latency over long distances (thousands of kilometers) and high availability using redundant network paths (rings). SONET (in the United States and some other countries) and SDH (in Europe and other parts of the world) are often referred to as SONET/SDH. SONET/SDH was designed to be a wide-area networking transport providing resilient, low-latency, high-bandwidth connections using fiber-optic networks and traditional electrical (copper) cabling. SONET and SDH networks can use WDM as an underlying infrastructure to maximize fiber-optic bandwidth and support additional network services.
While dedicated, dark fiber-optic service and WDM may not be available in all locations, SONET/SDH services are much more accessible--often from multiple service providers. Thus, SONET/SDH can be used as a local, metropolitan and global fiber-optic-based transport network for voice and data. As a network transport, SONET/SDH supports diverse network traffic, including TCP/IP, FC via generic framing protocol, asynchronous transfer mode (ATM) and packet over SONET (POS) among others.
SONET bandwidth is subdivided using time division multiplexing (TDM). SONET network bandwidth is allocated as optical carrier (OC) levels with SDH network bandwidth being allocated as a synchronous transport module (STM) level. OC levels are based upon multiples of 51.840Mb/sec; for example, OC-3 is 3 x 51.840Mb/sec or 155.5Mb/sec. Other bandwidth increments include OC-12 (12 x 51.840Mb/sec) and OC-48 (48 x 51.840Mb/sec). As a general guide, SONET/SDH latency delay is approximately one millisecond per 100 route miles (one-way) regardless of the bandwidth level. A route mile refers to the actual path the optical network takes between two points. Check with your bandwidth service providers as to what the route path and network latency will be for your specific application and get an SLA for that level of service.
Wavelength services: Carriers and service providers offer fiber-optic wavelength services that are a wavelength provisioned from a WDM device. A wavelength service is similar to a dark fiber, and can be used to span distances for private LAN and SAN network interfaces at various speeds. As opposed to dark fiber, with a wavelength service you have a private wavelength isolated, but share the physical cable with others. Wavelengths offer the fastest connection speeds, lowest latency and most effective bandwidth alternative when dedicated dark fiber isn't available. A wavelength service can be dedicated to a particular network transport (ESCON, Ethernet, FC or FICON) or for shared access using a SONET/SDH gateway. Wavelength services are considered managed solutions, and can be protected (built-in redundancy) or unprotected (no alternate paths). Users can attach their LAN and SAN network devices, including switches and routers, to the wavelength fiber-optic connection.
TCP/IP (IP): TCP/IP (IP) utilizes different transport networks and services, including SONET/SDH, WDM, and Ethernet on copper and fiber-optic cabling. Storage over IP can involve many different protocols and technologies, including NAS for file-based access of data, iSCSI for block data access, as well as FC transported over IP using iFCP or FCIP for long-distance SAN extension and remote mirroring.
IP is a good, general-purpose network protocol for storage apps that aren't sensitive to latency or that have lower bandwidth requirements, and for cost-sensitive storage environments. This isn't to say IP doesn't support high bandwidth--quite the contrary. IP supports high bandwidth when using fast network transports, including SONET/SDH, DWDM and carrier Ethernet. A common mistake is to look at dollar per Gb/sec and assume full bandwidth will be achievable. This is similar to evaluating disk storage simply on a dollar-per-gigabyte basis instead of a more realistic dollar per usable, effective gigabyte. When making comparisons among bandwidth speeds, an important number to compare is the packet delivery ratio (PDR). As more layers are involved in the movement of data, more latency will be introduced. The PDR number describes how high or low the packet retransmission and packet loss is, an indicator of effectiveness and latency. For storage apps, low latency can be as important as the amount of effective available bandwidth, particularly for time-sensitive, synchronous applications (see "Transport options: Pros and cons," this page).
|Transport options: Pros and cons|
Layering and latency
Remember that the more layers and protocols used in the network, the longer the latency and potential disruption to the storage applications. For example, on the left-hand side of the "Bandwidth layers" diagram, an IP-based network may have a lower cost for a specified amount of bandwidth. However, when looking at the multiple layers involved (FC mapped onto IP using FCIP, while the TCP and IP layers are mapped onto SONET), the latency adds up; depending upon the design of the network, congestion could reduce the effective bandwidth. Moving to the right on the diagram, an example of FC extension using ATM/POS mapped to SONET is shown, followed by FC mapped to SONET with less latency and layers, followed by FC on dedicated fiber optic (WDM). Understanding what layers are involved in the network is important because each layer adds complexity, cost and latency.
Reliability, availability and serviceability
Diverse network paths are critical for uninterrupted network service. Make sure your network provider can guarantee diverse network paths not only through its core networks, but its partners' core networks. Secondly, determine how the service provider will manage and guarantee network performance (low latency and effective bandwidth). It's possible that different network bandwidth providers could use the same network route paths. Fiber-optic bandwidth services, including wavelength services, can be protected or unprotected. With a protected optical or wavelength service, the service provider uses dual paths through its network. A protected bandwidth service may have a handoff (connection) to the user of a single or dual pair of fiber-optic cables depending on service type. Another alternative is to use self protection, where two diverse wavelengths of fiber-optic services are used from the same or different providers, each with a diverse path. In this scenario, the customer takes responsibility for integrating the fiber-optic network into their environment.
|A sampling of service providers|
Where to find fiber-optic services
There are many sources for different types of fiber-based services that vary by geographical location. When talking with prospective fiber-optic bandwidth service providers, ask about estimated and measured latency, as well as service level agreements between locations. If you're looking at wavelength services, check to see if there are any service provider restrictions on using TDM equipment for aggregation of slower and underutilized interfaces. Also verify with prospective service providers what they classify as a wavelength service, as some may consider a SONET OC-48 or OC-96 to meet that criteria, while others provide a true WDM-based lambda as a wavelength.
Some providers (see "A Sampling of service providers," this page) offer metropolitan services, others offer regional, while still others offer national and international services. Service providers will often have a map on their Web site showing coverage areas and types of services. Pricing for fiber-optic-based services can vary greatly depending on location, distance, type of service (protected or unprotected), amount of bandwidth, and managed or wavelength service. With bandwidth, you get what you pay for, so look at your application needs and match the appropriate level of capability to the specific requirement.
A common mistake is to look at bandwidth simply in terms of dollars per Gb/sec. The effective or actual usage amount is important, and with bandwidth that includes what level of utilization at a given response time (latency level) can be maintained without congestion and packet delay/loss. Some vendors estimate TCO savings of 25% to 30% (or more) when using a premium optical-based service, including SONET/SDH vs. an equivalent IP-based service, considering effective bandwidth, lower latency, less complexity and application improvements.
Don't make the mistake of prototyping a storage application at a reduced workload, and then assume that heavier workloads will scale linearly with regards to bandwidth and latency. Instead of linear scaling, effective bandwidth can drop off as workload is added along with additional latency, resulting in poor performance, particularly for synchronous-based storage applications. Test realistic workloads using different network bandwidth transport options to determine which ones meet your production needs and budget requirements. Pay close attention to details, including protected or unprotected services, diverse paths, problem resolution and network latency. Get various groups in your organization (storage, telecom and networking) to work with each other. First and foremost, understand your needs and the capabilities of these different technologies. Then align the appropriate technology to the specific task, application and cost of service.