By David Jacobs
The current economic climate has limited customer spending to projects that demonstrate measureable savings. 10 GbE makes possible reducing the number of port and switches required, thereby simplifying the task of combining multiple small data centers into large data centers.
Building effective consolidated data centers requires a careful analysis of requirements and a detailed understanding of network interface, switch and cabling technology. Working with customers as they plan and implement a combined data center provides multiple opportunities for VARs and integrators to provide services and sell equipment.
The following articles provide an overview of the issues involved in selecting equipment and upgrading customer network infrastructure to support the use of 10 GbE in a consolidated data center.
IN THIS GUIDE:
- Upgrade customer data centers with 10 gigabit Ethernet
- Investigating 10 gigabit Ethernet interconnect solutions
- Upgrading customer network fabric for 10 gigabit Ethernet
UPGRADE CUSTOMER DATA CENTERS WITH 10 GIGABIT ETHERNET
Cost-effective 10 gigabit Ethernet interfaces and switches provide opportunities for VARs and integrators to assist customers in reducing costs by consolidating data centers. Combining multiple small data centers into a single large center is a complicated process. Extensive analysis and planning are required before any new equipment can be purchased and installed.
Channel partners are uniquely qualified to guide customers through the upgrade process, but they must keep up to date. The technology is new and constantly evolving. Not all standards for 10 gigabit Ethernet (GbE) are stable. Partners must remain in touch with vendors and track vendor timetables to understand limitations in current product versions and the status of updates.
Consolidated data centers require increased data rates
Eliminating individual servers with virtual servers executing on a single higher-power server reduces equipment costs, power usage and administrative expense. Combining processing on a single server means combining the data traffic. While a 1 gigabit Ethernet link may have been sufficient for each server prior to consolidation, it is unlikely to be capable of supporting multiple virtual servers.
Reducing data center interconnect costs
Configuring one or a small number of 10 GbE links is clearly more cost-effective than a larger number of 1 GbE links. However, the benefits of 10 GbE extend beyond simply reducing the number of server and switch ports. 10 GbE offers the possibility of combining network and storage switches.
In the past, the throughput rates required to connect servers to storage area networks (SANs) and network-attached storage (NAS) have exceeded Ethernet capacity. As a result, these connections have utilized Fibre Channel or InfiniBand. The increase in throughput to 10 gigabits makes Ethernet competitive for server-to-storage connections.
Replacing separate switches for the network and for storage with a single 10 GbE switch will reduce equipment cost and power usage, but channel partners should exercise caution before urging customers to move quickly to make this change.
Simply increasing Ethernet to 10 gigabits is not sufficient to replace existing technologies. Storage interconnects cannot tolerate packet loss, but traditional Ethernet does not guarantee that packets will not be lost. Higher-layer protocols such as TCP detect and retransmit in the event of packet loss, but the time delay required is unacceptable for adequate storage performance.
The IEEE has developed a series of new protocols to address this problem. Successive revisions of these standards have been issued, but none has reached final ratification. All of the major switch vendors have developed implementations of these protocols, but tests at the University of New Hampshire Interoperability Lab earlier this year revealed incompatibilities among vendor implementations.
Inter-vendor compatibility cannot be expected until protocols are finalized and further rounds of tests completed. Customers who currently rely on a single vendor can proceed but should remember that not all vendor-announced capabilities are actually available in currently shipping products.
Proceed with caution. Storage throughput and latency are critical. Establish a baseline before introducing new interfaces and switches. Compare the baseline against measured performance after each change, and be prepared to work with equipment vendors if problems occur.
Server interfaces continue to evolve
Server interface technology is an additional area where channel partners should monitor vendor plans. Integrating the network interface onto the server motherboard reduces the overall solution cost. Motherboard interfaces for each successive Ethernet generation have become available soon after the technology matured.
A possible mismatch is looming between the product plans of interface chip vendors and switch vendors. The major chip vendors have announced interfaces for 10GBase-T, the 10 GbE twisted-pair standard, while switch vendors are promoting SFP+ owing to its low cost, reduced latency and power requirements compared with 10GBase-T. SFP+ limits interconnect distance to 10 meters, which is more than sufficient to connect the server to a top-of-rack switch, according to Dave Passmore, research director at the Burton Group. Passmore has observed that some customers are finding it difficult to plan, but the situation may change when the first 10GBase-T LAN on motherboard products ship in late 2010 or early 2011.
Data center update requires changes in management software and practices
Combining network and storage interconnects in a single set of switches requires changes in both management software and practices. What had been two separate sets of tasks assigned to distinct individuals now merges. The change requires staff members to add new areas of expertise as well as learn the mechanics of new software packages.
Partners should be aware of potential pushback from data and network center staffs. Combining functions may be seen as a threat by some individuals. Staff issues can undermine a project as effectively as technical problems. Partners should work with management to anticipate these types of problems early in the project.
Despite potential difficulties, channel partners should not hesitate to recommend that customers utilize 10 GbE technology. Though still evolving, the new interfaces, switches and protocols offer significant opportunities for performance increases and cost reduction.
INVESTIGATING 10 GIGABIT ETHERNET INTERCONNECT SOLUTIONS
A wide variety of cable and interconnect types is available to support 10 gigabit Ethernet (10 GbE). Solutions vary in terms of maximum interconnect distance, power and heat, signal latency, reliability, and adaptability to future requirements. Total cost includes more than simply the cost of equipment interfaces and cables. Labor is often a major factor. Choosing a solution requires carefully evaluating each option against your application requirements.
The basic choice is between optical fiber and copper. Until recently, 10 gigabit rates could be achieved only by using optical fiber, but now standards have been established permitting the use of twisted pair. Fiber and copper can be used in the same data center with the choice for each connection based on the distance between devices to be connected. To further complicate the decision, several options are available for both fiber and copper.
Fiber options include multi-mode or single-mode fiber. Multi-mode is limited to shorter distances than single-mode. Because multi-mode cable is thicker than single-mode, portions of the light beam follow different paths as they bounce back and forth between the walls of the fiber. The result is that the signal reaching the other end of the cable is distorted. The amount of distortion increases with the length of the cable. The light beam follows a single path through thinner single-mode cable, so the amount of distortion is much lower.
Multi-mode cable was installed in the 1980s to support FDDI at 100 Mbps. This cable was 62.5 microns thick. The initial 10GBase-SR fiber interface standard was limited to 26-meter connection distances using this cable. Removing and replacing installed cable is expensive, so the IEEE developed 10GBase-LRM, also known as IEEE 802.3aq. It uses a circuit that compensates for distortion. The result is that the new standard can drive signals 220 meters over 62.5 micron cable.
Cable production techniques have improved. Newly manufactured multi-mode cable is only 50 microns thick. The 10GBase-SR interface can support 300-meter connection distances with new, thinner multi-mode cable.
Single-mode fiber can carry signals up to 40 km, so it is used to connect to wide-area networks. It is more difficult to connect because of the thinness of the cable, and it requires a more expensive laser light source than multi-mode does. The result is that multi-mode is used where the connection length does not require single-mode.
10GBase-CX4, standardized by the IEEE as 802.3ak in 2002, was the first available copper interconnect for 10 GbE. Similar to copper InfiniBand cable, 10GBase-CX4 uses four cables, each carrying two-and-a-half gigabits of data. It is limited to approximately 10 to 15 meters but provides an extremely cost-effective method to connect equipment within that distance.
Small Form-Factor Pluggable Copper (SFP+CU) provides another low-cost, low-energy interconnect. It is limited to approximately 10 meters.
IEEE 802.3an, more widely known as 10GBase-T, was completed in 2006 and is expected to become the most widely used 10 GbE interconnect method. It utilizes familiar twisted pair cable and RJ-45 connectors.
Current interface chips consume more power, produce more heat and introduce greater latency than fiber or earlier copper interfaces. The same problems were encountered with initial Gigabit Ethernet twisted pair interfaces, so it is expected that second (and later) generation chips will improve. In the meantime, these issues have slowed 10GBase-T adoption.
In many cases, existing twisted pair cable must be replaced. Although some installations have been successful running 10 GbE over installed Category 5 cable, this is not recommended. Even if testing indicates that the cable can carry 10 megabit traffic, intermittent problems may occur in the future.
Category 6, 6a UTP or 6a F/UTP (foil wrapped UTP) is recommended. Category 6 cable is limited to 55 meters, while 6a or 6a F/UTP can support 100-meter connections. Category 6 and 6a cables are thicker than Category 5 to reduce the possibility of alien crosstalk -- that is, crosstalk between adjacent cables. Thicker cable means that conductors in adjacent cables are farther apart, reducing the possibility of crosstalk. Foil wrapping virtually eliminates the possibility of alien crosstalk for Category 6a F/UTP.
Replacing existing cable may not be as simple as pulling out one cable and putting in another. Increased cable thickness means that Category 6 cable takes up more space in cable trays. Additional trays may be necessary if there isn't enough room in the existing tray. Care must also be taken to avoid piling too many cables on top of one another. The weight can squeeze cables on the bottom and reduce the spacing between conductors.
Evaluate labor costs carefully. Fiber has been considered more expensive to install because care must be taken to avoid tight bends and install connectors. Twisted pair at 10 gigabit rates also requires extra care, compared with lower-bandwidth twisted pair, when attaching connectors and arranging cable in trays. Installation costs for twisted pair can increase significantly if additional trays must be installed.
Plan for the higher data rates
100 GbE is on the way. The IEEE 802.3ba committee began working on 100 GbE in 2007 and expects to complete the work in 2010. The goal is to support 40-km connections over single-mode fiber, 100 meters over multi-mode, but only 10 meters over copper. Consider installing fiber now, even if copper appears the best choice for current needs.
UPGRADING CUSTOMER NETWORK FABRIC FOR 10 GIGABIT ETHERNET
Upgrading network fabric to support 10 gigabit Ethernet (GbE) requires a careful analysis of future as well as current requirements. Combining the network upgrade with consolidation of multiple smaller data centers into a single large data center greatly increases the complexity of the process. VARs and solution providers are well positioned to assist customers, beginning with analysis, through cabling and equipment selection and purchase, installation, configuration and testing.
Data center consolidation drives network redesign
Data center consolidation requires extensive network redesign. Replacing individual servers in remote sites with a smaller number of higher-capacity servers in a central data center requires adding high-throughput WAN links from the remote sites to the consolidated data center.
Within the central data center, multiple virtual servers executing on a single physical server generate higher data rates than would typically be generated by a single server. In many cases, this will mean replacing 1-GbE links with 10 GbE. Network cabling and switches must be upgraded and in many cases replaced. Combining network and storage traffic on a single set of interconnects further increases throughput requirements and adds complexity to the design task.
Network simulation and planning software can aid the design process. Channel partners can recommend a planning tool from commercial vendors such as Opnet Technologies or Shunra Software or assist the customer in using an open-source tool such as NS-2 or OMNet++.
Most planning tools create detailed network documentation. If the customer does not have processes in place to maintain up-to-date network documentation, the major redesign period is a good opportunity to begin. Work with the customer to create a network update process that ensures that future changes will be documented.
Selecting a cable technology to support 10 GbE
A variety of interface and cable types, both copper and optical fiber, support 10 GbE. The choice between copper and fiber is dictated mainly by interconnect distance, but consider also interface power usage and packet latency. Customer experience and familiarity with each type of cable can be factors in the decision.
Cable types can be mixed in the same data center. For example SFP+ copper can be used for the short run from a server to a top-of-rack switch, with fiber used for longer interconnects.
Whichever cable technology is chosen, careful attention must be given to cable placement. Twisted pair cable to support 10 GbE is thicker and heavier than twisted pair sufficient for lower data rates. It is not always possible to simply replace the cable in existing trays. Additional trays may be needed and existing trays moved.
Problems with thickness or weight are unlikely when replacing twisted pair with fiber or replacing multi-mode fiber with single-mode, but pay close attention to fiber minimum bend radius. Paths that may have been acceptable for twisted pair may not meet minimum radius requirements for fiber.
It is critical to use compatible cabling components -- including patch panels, patch cords and cables -- from end to end, says Stephan Fowler, infrastructure engineer with Celergy Networks Inc., a California-based network integrator.
Some vendors manufacture all of these components, while others have formed partnerships to produce and test compatible components. Individual components from various manufacturers may be rated at 10 gigabits, but the end-to-end network may not support this rate if components that have not been tested together are used.
Upgrade switches for 10 GbE
All of the major switch vendors have released products supporting 10 GbE, and most have announced support for Converged Enhanced Ethernet. Examine vendor claims carefully. Some vendors have made more progress than others in implementing these protocols; announced support does not always translate to availability on a shipping product.
Suggest that customers examine even lower rate switches as part of the overall network redesign. Many switch ports that previously supported desktop users may be unused as a result of staff reductions or a switch to WLANs. Power usage and rack space can be reduced by eliminating unneeded switches.
Consider next year's requirements
Finally, consider possible future applications in the plan. There may be no current plan for VoIP, but planning for it now may be simpler than retrofitting it on the network later.
Consider also that data rates will continue to increase. The IEEE 802.3ba Committee expects to complete standards for 40- and 100-gigabit Ethernet in 2010, and vendors have announced plans to support these rates. Consider next-generation data rates when choosing cable technology and switches.
About the author:
David B. Jacobs of The Jacobs Group has more than 20 years of networking industry experience. He has managed leading-edge software development projects and consulted to Fortune 500 companies as well as software startups.