Académique Documents
Professionnel Documents
Culture Documents
Abstract
The Fibre Channel (T11.3) standards committee
developed a Standard called Fibre Channel over
Ethernet (FCoE)
The FCoE standard specifies the encapsulation of
Fibre Channel frames into Ethernet Frames and the
amalgamation of these technologies into a network
fabric that can support Fibre Channel protocols and
other protocols such as TCP/IP, UDP/IP etc.
A Direct End-to-End FCoE variant has been
accepted for the next version of the Standard
The tutorial will show the Fundamentals of these
FCoE concepts and describe how they might be
exploited in a Data Center environment
Fibre Channel over Ethernet (FCoE)
2010 Storage Networking Industry Association. All Rights Reserved.
Agenda
Introduction
FCoE Fabrics & Convergence
Architecture
Discovery & Link Instantiation
With FCFs
Direct End-to-End (w/o FCFs)
Introduction
This presentation provides an overview of Fibre Channel over
Ethernet (FCoE)
One should think about FCoE as placing the FC protocol on a new
physical link
New Lossless Ethernet links instead of physical FC links
But it is still Fibre Channel
The protocol has been defined in the INCITS Fibre Channel (T11.3)
technical committee
The needed Lossless Ethernet links has been defined in the IEEE
802.1
A newly defined Direct End-to-End (VN2VN) protocol will also
be explained
Fibre Channel over Ethernet (FCoE)
2010 Storage Networking Industry Association. All Rights Reserved.
FCoE Fabrics
&
Convergence
Fibre Channel over Ethernet (FCoE)
2010 Storage Networking Industry Association. All Rights Reserved.
FCoE Fabrics
(part 1)
(This is NOT Traditional Ethernet)
Lossless switches and fabrics (e.g., supporting IEEE 802.3 PAUSE) configurations are required
Jumbo frame support is strongly recommended (not a standard, but widely available)
FC
FC
Connections to an Integrated
CEE/DCB - FCoE Switch
Fibre Channel is carried over lossless Ethernet as a L3 protocol
Customer Applications
SCSI
Network Applications iSCSI
UDP
TCP
IP
Fibre Channel
FCoE
(FCoE VN_Port)
Ethernet port with IP & FCoE VF_Port
capabilities
FCoE Fabrics
(part 2)
(This is NOT Traditional Ethernet)
CEE/DCB only Ethernet switches may also exist in an FCoE capable Fabric but
one or more switches with FCoE capabilities must also exist
FCoE fabrics must inter-operate seamlessly with real FC Fabrics
FC services must operate identically on FCoE fabrics and Fibre Channel fabrics
FCoE must support all Fibre Channel advanced features (e.g. virtual fabrics, IFR,
security, etc.) transparently
FCoE will not require changes to FC software (Apps, Drivers, etc.)
However, vendors will enhance Drivers & Mgnt to exploit new capabilities
10
OS3
DB Server
OS2
App Server
Hyper Visor
OS2
App Server
FC
HBA
OS1
Web Server
Messaging
FC SAN
Ethernet
TCP/IP
E-HBA
(CEE)
MPI
RDMA
IB/Ethernet
Cluster
Today
Data Center
CEE/DCB
Network
OS2
App Server
OS3
DB Server
E-HBA
(CEE)
OS1
Web Server
OS3
DB Server
NIC
TCP acceleration.
MPI, RDMA
over Ethernet
FCoE
With
CEE/DCB
E-HBA
(CEE)
Data Center
CEE/DCB
Network
11
CEE/DCB-FCoE
Ethernet SW
FC
FC & CEE/DCB
Ethernet
SW Blades
FC
FCoE
Outfacing
IP Network
Data Center
Fabric
(LAN/WAN)
Including iSCSI
Gateways
Messaging
NAS
FICON
FC
FICON Storage
Controller
CEE/DCB Ethernet
SW
File Storage
Arrays (NAS)
Business
Campus
with iSCSI
connections
FC Link
FICON
Link
Ethernet
Link
12
Architecture
13
FC-4
FC-3
FC-3
FC-2
Unchanged
FC Levels
FC-2V
FC-2V
FC-2M
FC-2P
FCoE Mapping
FC-1
MAC
FC-0
PHY
(Located in the
FC Entity*)
(Located in the
FCoE Entity*)
IEEE 802.3
Layers
* Discussed in later slides
Ethertype
FCoE
(8906h)
FCoE
Header
FC Header
SCSI Commands/Data
FCS
Ethertype
FIP
(8914h)
FIP
Header
Descriptors
FCS
14
FCFs Discover each other, & VLANs (if any) then form a Fabric
ENodes Discover VLANs (if any) & then ENodes & FCFs Discover
Potential VN_Port VF_Port pairing
Capabilities of Potential pairing
Login Phase
FCoE
Initialization
Protocol
(FIP)
FCoE
Protocol
15
31-24
23-16
15-8
7-0
1
Optional IEEE 802.1q
4 Byte
Tag goes here
2
3
Reserved
Reserved
6
This field varies
In size
n
Ethernet frame
Size is 64 Bytes to
2220Bytes
Ver (4b)
n+1
n+2
Reserved
Reserved
(12 bits)
SOF (8 bits)
Encapsulated FC Frame
FC Frame = Minimum 28 Bytes (7 Words)
Maximum 2180 Bytes (545 Words)
(including FC-CRC)
EOF (8 bits)
Reserved
Ethernet FCS
16
31-24
23-16
15-8
7-0
1
Optional IEEE 802.1q
4 Byte
Tag goes here
2
3
4
See Appendix
Below for
Descriptor list items
Descriptor list
varies
In size
Reserved
FIP SubCode
Reserved
FIPLength
subcode
Descriptor List
Flags
Descriptor List
FP SP
(12 bits)
Descriptor
List Length
Flags
Ethernet frame
n+1
Ethernet FCS
size
Is 64Bytes to 2220Bytes Capability Bits (SPMA or FPMA)
Available bit
Fibre Channel over Ethernet (FCoE)
2010 Storage Networking Industry Association. All Rights Reserved.
ASF
F S S F
P P
Solicited bit
FCF bit
17
FCF Model
OR
Link
End
Point
(LEP)
OR
18
FC-3 /FC-4s
FC-3 /FC-4s
VN_Port
FC
Entity
VN_Port
FC
Entity
FCoE_LEP
FCoE
Entity
FCoE_LEP
FCoE
Entity
...
FCoE
Controller
Ethernet_Port
Burnt-in
MAC Address
In this
model this
is where
FC-2V
functions
live
In this model
this is where
the
Encapsulation
/DeEncapsulation
functions live
MAC Address of
FCoE_LEP
(VN_Port)
With FPMA it is
different than that
of the FCoE
controller
19
Examples of
single MACs
with
connections
to two
different
FCFs
FCoE_LEPs
FCF-MAC(A)
MAC(H1)
FCF A
VF_Port
H1
FCoE_LEPs
FC
Fabric
VN_Port
FCoE_LEP
Lossless
Ethernet
Switch
VN_Port
FCoE_LEP
H2
FCF-MAC(B)
VF_Port
FCoE_LEPs
FCF B
VF_Port
FCoE_LEPs
MAC(H2)
VF_Port,
VF_Port_Name
VN_Port
Traditional
Ethernet
LAN
Note: VF ports get created off of the FLOGI from the VN port and multiple VF ports can sit behind a single
FCF physical interface with a single MAC address
Fibre Channel over Ethernet (FCoE)
2010 Storage Networking Industry Association. All Rights Reserved.
20
B C
Host PCIe interface
FCoE
ASIC
External
Port
FCoE
function
NIC Function
Lossless
Ethernet
MAC
FC
Function
21
The FCoE controller will perform the FIP functions and will instantiate new
VN_Ports as FCoE Link End Points (LEP)
With a new MAC address specified by the FCF (FPMA)
A
FCoE Chip
MAC
Address of
Burnt-in
MAC
FC
Entity
FCoE
Controller
FC
Entity
FC-3 /FC-4s
VN_Port
FCoE
Entity
Lossless Ethernet
MAC Ethernet_Port
FCoE_LEP
Etype =
FCoE
or
FIP?
FC-3 /FC-4s
VN_Port
FCoE
FCoE_LEP
Entity
NIC
MAC
Address of
Burnt-in
MAC
FCoE
Function
FC Function
Used to separate HW
based FCoE from other
Ethernet Traffic
Most NICs come with
several Burnt-in MAC
Addresses
22
23
End-Node
FCF
FCoE Initialization
Protocol (FIP)
Discovery
Discovery
Phase
FC Command
(Using the FCF
selected MAC
Address as the SA)
Login Phase
Normal FC
Processing
FC Command responses
FCoE Protocol
See Appendix for more details on Discovery Consideration and Actions, including discovering VLANs
Fibre Channel over Ethernet (FCoE)
2010 Storage Networking Industry Association. All Rights Reserved.
25
CNA &
VN_Port (X)
(L1)
(L2)
(L3)
DCB/CEE Ethernet
Switch (B)
(L11)
(L5)
(L4)
(2)
(L8)
(L6)
Storage
Controllers
Host
Systems
(1)
(L7)
CNA &
VN_Port (Y)
(3)
(a)
(4)
(b)
(c)
(d)
A CNA to CNA FCoE path between these Switch ports is now also possible even
without an FCF using Direct VN2VN mode
DCB/CEE Ethernet
Switch (A)
CNA &
VN_Port (X)
CNA &
VN_Port (Y)
(L4)
(L1)
(L2)
(L3)
Storage
Controllers
Host
Systems
(1)
(2)
(a)
(b)
Additionally an internal CNA to CNA FCoE path within a CEE/DCB Switch is also
possible even without an FCF using Direct VN2VN mode
Fibre Channel over Ethernet (FCoE)
2010 Storage Networking Industry Association. All Rights Reserved.
26
End-Node
FCoE Initialization
Protocol (FIP)
(after Randomly
computing ID)
(repeat
until no
conflicts)
Discovery
Phase
FLOGI
Login Phase
FC Command
Normal FC
Processing
FC Command responses
FCoE Protocol
Fibre Channel over Ethernet (FCoE)
2010 Storage Networking Industry Association. All Rights Reserved.
27
Host
Systems
Create
VN_Port
MAC Addr
Storage
Controllers
Send/Receive
FLOGI/FLOGI ACC &
then FCoE Frames
between VN_Ports
Host
Systems
Storage
Controllers
After IDs and Potential Partners (VN_Ports) are identified within the Level 2 Ethernet:
FLOGI FIP frames, & FCoE frames will be exchanged directly between the VN_Ports
After Link Instantiation the IDs will be Periodically Beaconed (Multicast)
Beaconing permits detection of link loss (via time-outs) & incorrect LAN joins
Fibre Channel over Ethernet (FCoE)
2010 Storage Networking Industry Association. All Rights Reserved.
28
29
FC
30
FCoE Switch
FC
Fabric
CEE/DCB
Network with all
CEE/DCB - FCoE
Switches
CEE/DCB
Network
FC
Fabric
CEE/DCB FCoE
Switch
CEE/DCB FCoE
Switch
CEE/DCB
Network
FCoE
Switch
CEE/DCB - FCoE
Switch
FCoE
Switch
CEE/DCB
Network
2
FC
Fabric
CEE/DCB
Network
1
FC
Fabric
FCoE
Switch
CEE/DCB
Network
4
31
Ethernet
FC
32
Internet
Classical
Ethernet
Network
CEE/DCB
Network
FCoE
Switches
FC
Fabric
CEE/DCB
Network
FCoE
Switch
FCoE Flows
IP Flows
Fibre Channel over Ethernet (FCoE)
2010 Storage Networking Industry Association. All Rights Reserved.
33
Internet
Classical
Ethernet
Network
CEE/DCB
Network
FCoE
Switches
FC Fabric
FCoE
Switch
CEE/DCB
Network
FCoE Flows
Invalid FCoE Flows
Fibre Channel over Ethernet (FCoE)
2010 Storage Networking Industry Association. All Rights Reserved.
34
Summary
35
FCoE Summary
T11.3s FC-BB-5 Ad-Hoc Working Group completed the specification in June 2009
FCoE is a simple, efficient mechanism for encapsulating Fibre Channel in Ethernet frames
on a New Ethernet type Network
Not a traditional Ethernet Interface or fabric
FCoE is made for a Data Center Fabric Not applicable for the Outfacing Network
T11.3s FC-BB-6 Ad-Hoc Working Group accepted VN2VN for the next specification
Look for VN2VN to provide viable FCoE SANs for the Entry/Low-End IT Environment
Will be seen to be competitive with iSCSI, especially with the SW version of FCoE
Expect CEE/DCB to be included in 1Gb Ethernet Switches
36
Q&A / Feedback
Please send any questions or comments on this
presentation to SNIA: tracknetworking@snia.org
Many thanks to the following individuals
for their contributions to this tutorial.
SNIA Education Committee
Claudio DeSanti
Robert Snively
Joe Pelissier
Howard Goldstein
Suresh Vobbilisetty
John Hufferd
Walter Dey
Silvano Gai
37
Thank You!
Appendix
Full ENode Model
(with VN2VN)
39
FC-3 /FC-4s
VN_Port FC-2V
FCoE_LEP
FCoE
Entity
VLAN-1
(Maybe the
default
VLAN)
VLAN-4
VLAN-3
VLAN-2
40
41
FCoE Fabrics
(part 4)
42
43
Flows
44
CEE/DCB
Network
FCoE
Switch
Ethernet
FC
H2
FCoE
Switch
#2
#1
CEE/DCB
Network
#3
CEE/DCB
Network
#4
FCoE
Switch
H4
H3
H1
FC
S2
CEE/DCB
Network
S1
Destination
& Source
Encapsulated
FC Frame
D_ID
S_ID
Path #1
FCoE-A MAC
FCoE-H2 MAC
FC_ID for S1
FC_ID for H2
Path #2
FCoE-B MAC
FCoE-A MAC
FC_ID for S1
FC_ID for H2
Path #3
FCoE-C MAC
FCoE-B MAC
FC_ID for S1
FC_ID for H2
Path #4
FCoE-S1 MAC
FCoE-C MAC
FC_ID for S1
FC_ID for H2
45
H2
Ethernet
Switch
Ethernet
Switch
#1
FCoE A
Switch
Ethernet
Switch
Lossless Ethernet
(CEE/DCB) Network
Path #1
FCoE-A MAC
FCoE-H2 MAC
Path #2
FCoE-C MAC
FCoE-A MAC
FC Fabric
#2
Ethernet
Switch
H1
Ethernet
Destination
& Source
FCoE
Switch
#3
Ethernet
Switch
S2
S1
Path #3
FCoE-S1 MAC
FCoE-C MAC
Encapsulated
FC Frame
D_ID
FC_ID for S1
FC_ID for S1
FC_ID for S1
S_ID
FC_ID for H2
FC_ID for H2
FC_ID for H2
Logical
Transaction Path
Ethernet
FC
46
S2
H2
Ethernet
Switch
#3
FC Fabric
#2
Lossless Ethernet
(CEE/DCB) Network
S3
Ethernet
Switch
S1
H1
Ethernet
Destination
& Source
#1
FCoE
Switch
C
Ethernet
Switch
FCoE
A
Switch
Ethernet
Switch
Ethernet
Switch
Path #3
Path #2
FCoE-S2 MAC
FCoE-A MAC
FCoE-A MAC
FCoE-C MAC
Path #1
Encapsulated
FC Frame
D_ID
FC_ID for S2
FC_ID for S2
FC_ID for S2
S_ID
FC_ID for H5
FC_ID for H5
FC_ID for H5
Logical
Transaction Path
Ethernet
FC
47
Additional Topologies
48
Additional Topologies
Lossless
Ethernet
(CEE/DCB)
Switch (X)
FCoE
Switch (A)
(1)
FCoE
Switch (E)
FC
FCoE
Switch (B)
Lossless
Ethernet
(CEE/DCB)
Switch (Y)
FCoE
Switch (F)
49
Additional Topologies
Lossless
Ethernet
Switch
(CEE/DCB)
FCoE
Switch (A)
(2)
FCoE
Switch (E)
FC
Lossless
Ethernet
Switch
(CEE/DCB)
FCoE
Switch (B)
FCoE
Switch (F)
50
51
FCoE Switch
FCoE Switch
52
FCoE Switch
FCoE Switch
53
Equivalent FC Topology
54
55
VLAN Determination
There may need to be a dynamic determination of what VLANs
can be used for FCoE operations
But this is optional -- the FIP VLAN discovery protocol is not needed if
these VLANs are already known or if VLANs are not used
Expect this to be administratively determined
56
Sometimes a New FCF will come on line and Multicast its availability to ALL_
ENode_MACs and All_FCF_MACs
The FIP Discovery phase exchanges solicitation and/or Advertisements between (HBAs
and/or FCFs) FCoE Controllers
Fibre Channel over Ethernet (FCoE)
2010 Storage Networking Industry Association. All Rights Reserved.
57
58
ENode Actions
When an ENode becomes operational:
The ENode discovers the VF_Port capable FCF-MACs
connected to the same Lossless Ethernet segment by:
Transmitting a multicast Solicitation to All-FCF-MACs (with the FCF bit
set to zero)
Receiving back Jumbo Unicast Advertisements from compatible VF_Port
capable FCF-MACs
(May store the discovered FCF-MACs in an FCF port list)
59
Timers connected to the ENodes & FCFs can determined that a port is
sending messages
Need to determine if a port is just dormant, but still alive
Therefore, a periodic message needs to be sent.
The periodic unsolicited Advertisements from the FCF can be used for
Keep Alive in the following directions
FCF ENode
FCF FCF
So -- a special FIP Keep Alive Message was created to just inform the
FCF that the ENode and its VN_Ports are still alive
Fibre Channel over Ethernet (FCoE)
2010 Storage Networking Industry Association. All Rights Reserved.
60
FC Switching Element
ENode-MAC
VN_Port
VN_Port
VN_Port
FCoE_LEP
FCoE_LEP
FCoE_LEP
FCoE
Controller
FCF-MAC
FC
FCoE
FCoE
Controller
VF_Port
FCoE_LEP
FCoE_LEP
FCoE_LEP
Ethernet
Ethernet_Port
Ethernet_Port
ENode-MAC
FCF-MAC
VN_Port1
MAC
VN_Port2
MAC
VN_Port3
MAC
Lossless
Ethernet
Switch
FCF-MAC
FCF-MAC
FCF-MAC
61
When intermediate switches see this frame they should cleanup their
ACLs etc.
Fibre Channel over Ethernet (FCoE)
2010 Storage Networking Industry Association. All Rights Reserved.
62
FIP Descriptors
63
Subcode
0001h
01h
Discovery, Solicitation
02h
Discovery, Advertisement
01h
FLOGI/FDISC/LOGO/ELP, Request
02h
FLOGI/FDISC/LOGO/ELP, Reply
01h
02h
01h
02h
FFF8h .. FFFEh
00h .. FFh
Vendor Specific
All others
All others
Reserved
0002h
0003h
0004h
Operation
64
Value
Type
Value
00
Reserved
09
01
Priority
10 (0Ah)
ELP
02
MAC Address
11 (0Bh)
VN_Port ID
03
FC-MAP
12 (0Ch)
FKA_ADV_Period
04
Node_Name /
Switch_Name
13 (0Dh)
Vendor ID
05
Fabric
14 (0Eh)
VLAN
06
Reserved
07
FLOGI
Vendor Specific
08
NPIV FDISK
255 (FFh)
Reserved
65
Type = 5
Reserved
Len = 4
VF_ID
FC_MAP
Fabric_Name
66
Len = 11/10/9
67
Type = 13
Len = 3
Reserved
Vendor_ID
Type = 14
Len = 1
Rsrvd
FCoE VID
68
VLAN Request
ENode/FCF FCF
FIP Operation Code = 0004h
Descriptor List Length = 2-4
Type = 2
Reserved
VLAN Notification
FCF FCF/ENode
SubCode = 01h
FLAGS
Length = 2
Length = 3
SubCode = 02h
FLAGS
Length = 2
ENode/FCF-MAC Address
Type = 4
Reserved
FCF-MAC Address
Reserved
Type = 14
Len = 1
Rsrvd
FCoE VID -1
Type = 14
Len = 1
Rsrvd
FCoE VID-n
69
Type = 4
SubCode = 01h
FLAGS
Type = 3
Length = 2
Reserved
Type = 4
FC-MAP
Length = 3
Reserved
VF_ID
FC_MAP
Fabric_Name
Type = 12
Reserved
Len = 4
Reserved
Length = 2
FCF-MAC Address
Reserved
Switch_Name
Type = 5
Len = 3
Len = 2
Reserved
FKA_ADV_Period
Padding to Max Receive Size of Soliciting Entity, if
Solicited (i.e. if S=1b, otherwise no padding
Switch Name
Type = 6
Length = 1
70
FIP FLOGI
71
72
73
FIP ELP
74
ENode FCF
(The Keep Alive from the FCF ENode/FCF is a normal unsolicited Advertisement)
Fibre Channel over Ethernet (FCoE)
2010 Storage Networking Industry Association. All Rights Reserved.
75
FFFFFDh
( Fabric Controller
)
FFFFFDh
(Fabric Controller)
FCF FCF
FCF ENODE
(Note: the ENode uses FIP Fabric LOGO to Clear the Virtual Link from its side)
Fibre Channel over Ethernet (FCoE)
2010 Storage Networking Industry Association. All Rights Reserved.
76
77
78
Increasing Performance
10Gbps
The technology has evolved continuously, showing a great ability to adapt to new
technologies and increasing business requirements
Fibre Channel over Ethernet (FCoE)
2010 Storage Networking Industry Association. All Rights Reserved.
79