Académique Documents
Professionnel Documents
Culture Documents
Cisco Confidential
I/O consolidation supports all three types of traffic onto a single network
Servers have a common interface adapter that supports all three types of traffic
Different fabrics (network, SAN, HPC) have different requirements => do we have the technology which can serve them all at once? If we have it => is the technology mature enough and affordable to be massively deployed?
Cisco Confidential
Processor Memory
Processor Memory
I/O
Storage
I/O
I/O
LAN
I/O Subsystem
Storage LAN
IPC
IPC
FC Traffic FC Traffic
CNA CNA
Cisco Confidential
Storage
Must follow the Fibre Channel model
(Inter-Process Communication)
IPC
Cisco Confidential
Before PCI-Express there was not enough I/O bandwidth in the servers It needs to be Ethernet, but
2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential
Enabling Technologies
Cisco Confidential
10
Cisco Confidential
11
Transmission Error
Very rare in the data center
Congestion
Most common cause
Cisco Confidential
12
STOP
Switch A
PAUSE
Queue Full
Switch B
The PAUSE operation is used to inhibit transmission of data frames for a specified period of time
13
Start Sending Stop Frames for This Frames Again Interval of Time
2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential
PAUSE Frame
14
A
R_RDY
Cisco Confidential
15
EtherType = 0x8808 means MAC Control Frame Pause_Time is the time the link needs to remain paused in Pause Quanta (512-bits time)
Pad 42 Bytes
CRC
Cisco Confidential
16
Cisco Confidential
17
a.k.a. PPP (Per Priority Pause) PFC enables PAUSE functionality per Ethernet priority
16
EtherType = IEEE 802.1Q
Priority CFI
12 Bits
VLAN ID
Cisco Confidential
18
Ethernet Link
Receive Queues
One Two Three Four Five
Two
Eight Priorities
Seven Eight
Six
Seven Eight
Switch A
2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential
Switch B
19
Class vector indicates for which priorities the frame carries valid Pause information There are eight Time fields, one per priority
Pad 28 Bytes
CRC
Cisco Confidential
20
Cisco Confidential
21
DCBX
Hop-by-hop negotiation for:
Priority Flow Control (PFC) Bandwidth management Applications Logical link-down
Based on LLDP (Link Level Discovery Protocol) Allows either full configuration or configuration checking
Link partners can choose supported features and willingness to accept configuration from peer Added reliable transport
Cisco Confidential
22
Bandwidth Management
IEEE 802.1Q defines priorities, but not a simple, effective, and consistent scheduling mechanism
Products typically implement some form of Deficit Weighted Round Robin (DWRR) Proposal for HW-efficient, two-level DWRR with strict priority support Standard track in IEEE 802.1Qaz
Configuration and interworking is problematic
Cisco Confidential
23
Priority Groups
LAN Priorities Are Assigned to Individual Traffic Classes
SAN
24
10 GE Link Realized Traffic Utilization (30%) (30%) HPC Traffic (30%) LAN Traffic (40%) Storage Traffic (30%) T2 (20%)
3 Gbs
4 Gbs
6 Gbs
(50%)
3 Gbs
3 Gbs
3 Gbs
(30%) T1
(30%) T3
T1
T2
T3
Cisco Confidential
26
And technically FCoE is an extension of Fibre Channel onto a Lossless Ethernet fabric
2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential
27
Ethernet Header
FCoE Header
FC Header
28
Cisco Confidential
29
Protocol Organization
FCoE itself FIP (FCoE Initialization Protocol)
It is used to discover the FC entities connected to an Ethernet cloud
It is used to carry most of the FC frames and all the SCSI traffic Uses Fabric Assigned MAC address (dynamic)
Two different Ethertypes Two different frame formats Both are defined in FC-BB-5
http://www.cisco.biz/en/US/prod/collateral/switches/ps9441/ps9670/white_paper_c11-560403.html 30 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential
PFC
ETS
DCBX
DCB
2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential
Technically Stable
31
Cisco Confidential
32
If someone says it does not, it means he did not read the standard
However, current implementations may be behind the standard and do not fully support it yet
StorageNewsletter.com., Exclusive Interview with Darren Thomas, Head of Dell Storage. June 29, 2010
Cisco Confidential
33
FC-BB-5 fully defined the way to transport Fibre Channel over Ethernet
FC-BB-6 is working on adding features and functionality
Cisco Confidential
34
QCN is a core-to-edge protocol to deal with persistent congestion situations in a Layer 2 network
H1
H2
QCN message
DA: H3 SA: H1 DA: H3 SA: H2
When congestion is detected the core switch samples some frames, swaps their MAC addresses, and sends notifications backward
35
Cisco Confidential
DA: FPMA(T2)
Cisco Confidential
36
What Does This Mean? Layer 2 of the Fibre Channel model maps to Layer 3 of the Ethernet model
FC-4 FC-3 FC-2V FC-4 FC-3 FC-2V FCoE happens here (including Multihop) QCN happens here FCoE Entity Layer 2 - MAC Layer 1 - PHY Layer 3 FC Levels (Unchanged)
FC-2
FC Model
37
TRILL defines an alternative way to Spanning Tree to forward Ethernet frames in an Ethernet network
Also supports multipathing Has nothing to do with congestion
Source: Mellor, Chris. DCB is Not Enough. The Register August 3, 2010
Cisco Confidential
38
What Does This Mean? Layer 2 of the Fibre Channel model maps to Layer 3 of the Ethernet model
FC-4 FC-3 FC-2V FC-4 FC-3 FC-2V FCoE happens here (including Multihop) TRILL happens here FCoE Entity Layer 2 - MAC Layer 1 - PHY Layer 3 FC Levels (Unchanged)
FC-2
FC Model
39
Cisco Confidential
40
Industrys First I/O Consolidation Virtualization Fabric for Enterprise Data Center
Expansion Modules
Fibre Channel
8 Ports 1/2/4G FC
Fibre Channel
FC + Ethernet
6 Ports 10GE/FCoE/DCE
Ethernet
Partners OS Mgmt
2006 Cisco Systems, Inc. All rights reserved.
2x10GE/DCE/FCoE
SW FCoE/DCE + 2x10GE
2x10GE
Cisco NX-OS Cisco Fabric Manager and Cisco Data Center Network Manager
Cisco Confidential
41
32 Fixed SFP+ Ports Line Rate Hardware Capable of 1/10 Gigabit Ethernet Traditional Ethernet or Fibre Channel over Ethernet L3 capable (post FCS) FabricPath and TRILL capable (post FCS) 40 GE ready
32x 1/10GE host interfaces; 8x 10GE on network interfaces 10GE interfaces support FCoE HW supports 1G but SW support in a post-FCS release
Can mix-and-match with existing GE and next-gen GE FEX in network topologies Host port-channel support ACL classification
Cisco Confidential
43
Unified Fabric
Initial Deployments
Ethernet/LAN Ethernet/LAN
Servers and FCoE targets are directly connected to the Nexus 5000 over 10Gig FCoE Native Ethernet LAN network and Native Fibre Channel network break off at the Nexus 5000 access layer
Unified Fabric
Ethernet/LAN Ethernet/LAN
Blade servers connect to Nexus 4000 over 10Gig FCoE Nexus 4000 is a FIPSnooping Bridge
Native Ethernet LAN network and Native Fibre Channel network break off at the Nexus 5000
Blade Chassis
Cisco Confidential
Unified Fabric
Ethernet/LAN Core Ethernet/LAN Core
Nexus 5000 Nexus 5000 FCF FCF FEX-2232 FEX-2232 vPC vPC
FEX-2232 FEX-2232
Ethernet LAN
Unified Fabric
FC FCoE
VN VF
VE VE
VE VE
VE_Ports are run between switches acting as Fibre Channel Forwarders (FCFs) VE_Ports are bound to the underlying 10G infrastructure
VF VN
VE_Ports can be bound to a single 10GE port VE_Ports can be bound to a port-channel interface consisting of multiple 10GE links
47
All above switches are Nexus 5X00 Series All above switches are Nexus 5X00 Series acting as an FCF acting as an FCF
2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential
FCoE Adapters
Cisco Confidential
48
LAN
10GbE
CNA
10GbEE
10GbEE
Link
PCIe
Ethernet Drivers
10GbE Ethernet
Link
Link
PCIe
PCIe
Ethernet Drivers
Operating System
2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential
Operating System
49
Cisco Confidential
50
Full height, full length QLogic 4Gb FC controller and drivers Intel Ethernet controller and drivers Windows, Linux, & Vmware (ESX 3.5U4 & 4.0) support
51
Cisco Confidential
Power ~7.4W (Dual Port with Optical SR) Low Profile Form Factor
No heat sink required
Cisco Confidential
52
Cisco Confidential
53
Cisco Confidential
54
Open-FCoE Software
HBA HBA Mgmt Plane Linux Kernel File System layers SCSI Layer HBA Driver HBA Linux Kernel File System layers SCSI Layer OpenFC Layer FCoE Layer Ethernet Driver Ethernet Net Device FCoE FCoE Mgmt Plane
Fibre
2006 Cisco Systems, Inc. All rights reserved.
Server
Cisco Confidential
Ethernet
Server
55
More information
Cisco Confidential
56
Standards Sites
More Information
http://ieee.org http://t11.org/fcoe
Case Studies http://www.cisco.com/en/US/products /ps9670/prod_case_studies_list.html Book I/O Consolidation in the Data Center
http://fcoe.com
Cisco Confidential
57
Thank You
Cisco Confidential
58
Cisco Confidential
59