10 Gigabit Ethernet and Its X Factors

1 Abstract
You have read so much about 10 Gigabit Ethernet (10 GbE hereafter), and you are finally convinced that this is the next big thing for your data center. You now need to get down to the nitty-gritty of the technology. Possibly you are trying to figure out which are the options for your next wiring closet upgrade project, or maybe you are considering 10 GbE to consolidate your next-generation MAN requirements, or perhaps you are a technology enthusiast who likes to stay ahead of the curve. If any of the above applies to you, then this paper will be a compass to navigate your way out of the labyrinth of 10 GbE technologies and standard port types. At the end of this reading, youll be able grasp differences, similarities and applications of the many 10 GbE pluggable modules on Cisco switches and routers.

10 Gigabit Ethernet in the Standards

10 Gigabit Ethernet was ratified as a standard by the IEEE 802.3ae Task Force in 2002. Although three years in the Internet age could classify this as an old standard, 10 GbE technologies are still in a profound phase of maturation, and the 10 GbE market is still in its infancy. Before diving into the details of pluggable optics, we start off by reviewing the 10 GbE standards and by examining the most relevant applications for the various 10 GbE standard port types.

2.1

10 GbE over fiber optics media: IEEE 802.3ae standard at a glance

First and foremost 10 GbE is still Ethernet, just much faster. Besides raising the speed bar to 10,000 Gb/s, the main objectives of the 802.3ae 10 GbE standard were to: preserve the Ethernet frame format, maintain the maximum and minimum rate size of the 802.3 standard and, because the transmission medium of choice is fiber optics, support only full-duplex operation (dropping the requirement for the CSMA/CD protocol). A big portion of the work done by the IEEE 802.3ae standard has been focused on defining the physical layer of 10 GbE.

Figure 1. IEEE 802.3ae concentrated most of the work around the Physical Layer

Alessandro Barbieri, Cisco Systems, Inc. August 2005

As a result of the standardization effort, four new optical interface types have been defined (or in IEEE jargon, four different Physical Medium Dependent sublayers, a.k.a. PMDs) to operate at various distances on both single mode and multimode fibers. In addition to these four PMDs the standard introduces two new families of physical layer specifications (a.k.a. PHYs in the IEEE lingo) to support LAN as well as WAN applications. Overall there are seven possible port types as shown in Figure 2.

Figure 2. The seven port types created by the IEEE 802.3ae standard

The next two sections will examine in detail the four PMDs and the two families of PHYs.

2.1.1

IEEE 802.3ae PMD sublayers

The proliferation of PMD sublayers promoted by the standard can sound confusing at first. Each PMD has different technical characteristics in order to support different fiber media and operating distances. The approach chosen by IEEE can be explained with the intent to offer the cheapest optical technology possible for a particular application: PMDs for single mode fiber only: o 10GBASE-L operates in the 1300 nm band to support distances up to 10 km (6.2 miles). 10GBASE-E operates in the 1550 nm band to reach up to 40 km (24.8 miles).

PMD for multimode fiber only: o 10GBASE-S, by leveraging low cost 850 nm laser technologies, can cover distances of 26 to 82 meters on legacy multimode fiber. With laser optimized multimode fibers (known as OM3 fibers), 10GBASE-S operates up to 300 m.

PMD for multimode and single mode fiber: o 10GBASE-LX4 adopts an array of four lasers, each transmitting at 3.125 Gb/s, and four receivers arranged in a wavelength division multiplexing (WDM) fashion. Working in the 1300 nm region this PMD can support distances of 300 m on legacy FDDI-grade multimode fiber as well as distances of 10 km on single mode fiber.

Alessandro Barbieri, Cisco Systems, Inc. August 2005

Figure 3 shows at a high level the fiber and the range supported by each PMD. For additional details on the various fiber types and distances supported, refer to Cisco official data sheets: http://cco/en/US/products/hw/modules/ps5251/products_data_sheet09186a008007cd00.html.

Single Mode PMD Type (nm) Multimode Multimode OM3 (2000 MHz*km) 10GBASE-E ITU-T G.652

Single Mode

10GBASE-L

10GBASE-L4

Multi Mode

10GBASE-S

26/82

300

10,000

40,000 Distance (m)

Figure 3. IEEE 802.3ae PMD classification by fiber and operating range

Alessandro Barbieri, Cisco Systems, Inc. August 2005

2.1.2

IEEE 802.3ae families of PHY

Two new physical layer specifications are part of the 10 GbE standard framework: LAN PHY and WAN PHY. In general the properties of the PHY are defined in the Physical Coding Sublayer (PCS) responsible for the encoding and decoding functions. There are three types of PCS sublayers: 10GBASE-X, 10GBASE-R and 10GBASE-W. The first two are part of the LAN PHY family while the latter represents WAN PHY. LAN PHY and WAN PHY differ in the type of framing and interface speed. Serial LAN PHY (10GBASER) adopts Ethernet framing and the data rate is 10.3125 Gb/s (the MAC runs at 10,000 Gb/s and by adding the coding redundancy of 64B/66B the effective line rate becomes 10,000 * 66 / 64 = 10,3125 Gb/s.). On the other hand WAN PHY wraps the 64B/66B encoded payload into a SONET concatenated STS-192c frame in order to generate a data rate of 9.953 Gb/s.

Figure 4. PHY families

So why do we need WAN PHY? The traditional optical transport infrastructure is based on the SONET/SDH protocols which operate at a speed of 9.953 Gb/s. LAN PHY has a line rate of 10.3125 Gb/s which does not match the speed of SONET/SDH, thus it cannot be transported as it is over wide area networks based on SONET/SDH. A mechanism to transport 10 GbE across wide area networks built around SONET/SDH was deemed required. WAN PHY is the IEEE answer to adapt 10 GbE data rate to the speed of SONET/SDH, the dominant technologies deployed in optical transport networks. The purpose of WAN-PHY is to render 10 GbE compatible with SONET STS-192c format and data rate, as defined by ANSI, as well as the SDH VC-4-64c container specified by ITU. WAN PHY is not strictly SONET compliant, but rather we can think of WAN PHY as a SONET-friendly variant of 10 GbE. The optical specifications as well as the timing and jitter requirements remain substantially different from the SONET/SDH protocols.

Alessandro Barbieri, Cisco Systems, Inc. August 2005

2.2

10 GbE over copper: IEEE 802.3ak standard at a glance

In 2004 the family of 10 GbE standard port types saw a new entry: 10GBASE-CX4. Ratified by the IEEE 802.3ak Task Force, this is the first 10 GbE specification based on a copper interface. The whole premise of CX4 is to address the market demand for very low cost 10 GbE links in those applications that do not need the reach of fiber optics media. 10GBASE-CX4 runs over four pairs of twinaxial copper wiring up to 15 meters and adopts the IBX4 connector standardized by the Infiniband Trade Association.

Figure 4. CX4 leverages Infiniband cabling and connector specifications

10GBASE-CX4 will not be the only standard for 10 GbE over copper for too long. In fact the IEEE Task Force 802.3an is currently working on the 10GBASE-T standard with the objective to support distances of 55 to 100 m on Cat6 UTP cabling and 100 m on Cat7 and Cat6e. This upcoming standard is expected to be ratified in July 2006.

Alessandro Barbieri, Cisco Systems, Inc. August 2005

IEEE 10 Gigabit Ethernet Applications

As previously noted, the principal reason to bear this many port types (see Figure 5) in the 10 GbE standards is to address in a cost-effective fashion a variety of applications. Each physical layer technology is meant to address specific market requirements and, within that context, deliver the cheapest technical solution.

Figure 5. IEEE 10 GbE Port Types (2005)

To help to identify the set of applications best suited for each 10 GbE port type, Figure 6 proposes a loose mapping of the most common 10 GbE applications, categorized by reach and availability of a particular transmission media, to specific port types. (Indeed port types are not necessarily restricted to the applications reported in the chart.)

Figure 6. IEEE 10 GbE port types positioning

10GBASE-SR and 10GBASE-CX4 are mostly used in data center or server farm applications, where the limited distance requirements and the flexible choice of transmission media available (copper/fiber) make inter-rack server-to-switch and switch-to-switch interconnects the ideal target for these technologies.

Alessandro Barbieri, Cisco Systems, Inc. August 2005

In the enterprise campus the flexibility of 10GBASE-LX4 in terms of supported fiber media and reach makes this technology arguably the most relevant in enterprise networks. Although not costoptimized for the data center, LX4 can be used in this multimode fiber-rich space as well. One can argue that LX4 is really the Swiss army knife 10 GbE technology for the enterprise network. With 10GBASE-LR and 10GBASE-ER we step into the range of applications requiring extended distances and single mode fiber only: campus backbones and metro applications. These two port types are the most relevant to service providers for a variety of deployments such as metro Ethernet service aggregation, DSL backhaul, or inter-POP connectivity. Service providers look at WAN PHY for inter-POP or inter-office applications where the requirement is to interface to an optical transport infrastructure based on SONET/SDH to cover long distances. 10GBASE-W is also often the only options to those enterprises seeking to lease from a carrier an OC-192 TDM circuit or an OC-192 lambda for WAN applications spanning countries or even continents. At this point, we are familiar with the standard classification of 10 GbE interfaces and we have a taste of the various applications targeted by each port type. We are now ready to introduce the three types of Cisco 10 GbE optical transceivers, which represent the real-world incarnation of the various port types examined thus far.

Alessandro Barbieri, Cisco Systems, Inc. August 2005

Cisco 10 Gigabit Ethernet Pluggable Interfaces

Virtually every 10 GbE port on Cisco switches and routers today is sold separately from the optical or copper physical interface. This way the end user can customize the 10 GbE port according to its application requirements by selecting different hot pluggable transceivers. Cisco 10 GbE transceivers are based on industry-wide standards (known as Multi-Source Agreements or MSAs). Cisco standardized on three different flavors of 10 GbE transceivers with distinct electrical and mechanical characteristics: Xenpak (www.xenpak.org) X2 (www.x2msa.org) XFP (www.xfpmsa.org )

Figure 7. The X form factor of Cisco 10 GbE

From an end user point of view the most relevant differences among Xenpak, X2 and XFP, are found in the mechanical dimensions, the breadth of supported port types and the array of 10G protocol supported (besides 10 GbE). Figure 8 offers a comprehensive overview of such differences.

Supported Features

XENPAK

X2

XFP

All IEEE port types Non IEEE port types Size (mm)

Y Y (80 km/DWDM) 126x36x17

Y N 100x36x12

No LX4 N 78x18x10

Connector type

SC

SC

LC Y (OC-192 9.95 Gb/s, G.709 10.709 Gb/s)

Protocols other than 10GbE

Figure 8. X form factors: a user perspective (status quo 2005)

Xenpak is the largest form factor of the three. But with size comes also a versatility advantage: more room on the module to integrate optical and electrical components as well as higher thermal efficiency which in

Alessandro Barbieri, Cisco Systems, Inc. August 2005

turn relaxes the requirement on power dissipation. Consequently Xenpak nowadays supports the deepest range of port types sporting even a power-hungry DWDM interface as well as an 80 km PMD. As technology matures and evolves the gap in terms of supported port types existing today is bound to disappear. X2 being electrically compatible with Xenpak allows sharing board designs and components with Xenpaks and at the same time enabling small switches with size and thermal constraints to support 10 GbE. XFP is the smallest of the pack partly because it removes much of the electronics from the module transferring the cost on the host line card. It is interesting to note how by design XFP also supports the telecom protocols, hence making this form factor an attractive option for routing platforms supporting Ethernet as well as Packet over SONET/SDH and RPR interfaces. Another relevant differentiation from an end user perspective is the hardware compatibility of the trio of X transceivers with Cisco switches and routers. The information in the following table (Figure 10) is subject to change as new line cards will be introduced on these platforms.

Catalyst 6500

Catalyst 4500/494810GE

Catalyst 3750

Cisco CRS-1

Shared Port Adapters (10G Routers)

Cisco 7600

Xenpak X2

x x

XFP

Figure 9. X form factors compatibility chart

Alessandro Barbieri, Cisco Systems, Inc. August 2005

Beyond the Standards 10 Gigabit Ethernet Port Types: DWDM and ZR Xenpaks

Cisco sports two non-IEEE Xenpak port types in its 10 GbE product line: DWDM (shipping) and ZR (soon to be released). The former implements a PMD to transmit 32 different channels on the same fiber for a whopping capacity of 320 Gb/s over a single strand of fiber. With the aid of optical amplifiers the DWDM signals can be carried for distances up to 200 km. The latter is capable of reaches in the 80 km range, doubling the distance specified by the 802.3ae 10GBASE-E standard. Cisco has been shipping DWDM Xenpak since July 2004. This device is and will be for quite some time the most complex and technology-rich pluggable module on the market. The last section is devoted to examine more in details the marketing drivers and applications for the DWDM Xenpak.

5.1

DWDM Xenpak: the quest for unlimited bandwidth

DWDM Xenpak allows customers to significantly reduce the amount of transport equipment in the network by eliminating the need to perform wavelength conversion using dedicated DWDM transponders (see Figure 10). But what is really unique about DWDM Xenpak is that it enables a seamless integration of the optical layer with switching and routing. With the DWDM Xenpak, 10 GbE routers and switches blur in the same platform and architecture the OSI functions of layer one through layer three and above. DWDM Xenpaks are offering a new level of integration between transport and Ethernet/IP functions on high-end switches and routers. Service providers are finding attractive this new paradigm of building next generation triple-play networks because, by consolidating the DWDM function and Layer 2/Layer 3 forwarding on the same platform, they incur in significant cost savings from a CapEx as well as OpEx standpoint.

Figure 10. Integrating WDM optics into a switch/router eliminates transponder from the network

Enterprise customers deploying 10 GbE DWDM are also attracted by this technology as it largely removes the need for dedicated transport equipment, which is sometimes too complex and expensive to operate for an enterprise. With DWDM Xenpaks the transport layer can be limited to simple passive multiplexer devices that dont require any management. Enterprise customers deploy DWDM Xenpaks even if for initial deployment when only one channel is required: the value of DWDM Xenpak lies in its ability to gracefully upgrade the bandwidth on the same fiber infrastructure by simply plugging new DWDM Xenpaks into the core switches. Whats more, this is accomplished by simply managing the optical layer from the familiar Cisco IOS CLI interface.

Alessandro Barbieri, Cisco Systems, Inc. August 2005

10