Networking Technologies Concepts Internal External Storage PDF

EDUCATION
Networking Technologies:
Concepts in Internal and
External Networked Storage
Howard Goldstein
Howard Goldstein Associates, Inc.
Howard.Goldstein@HGAI.Com
Networking Technologies: Concepts in Internal and External Networked Storage©

Copyright 2007 Howard Goldstein Associates, Inc. HGAI All Rights Reserved.
Storage Networking Concepts
EDUCATION
• Objectives:
– Identify what networking is and how it relates to
storage
– Examine storage networking technology similarities
and differences
– Clarify and explain the language of networking:
Terminology
– Uniquely inform and hopefully entertain
Networking
Technologies
Networking Technologies: Concepts in Internal and External Networked Storage 2

© Copyright 2007 Howard Goldstein Associates, Inc. HGAI All Rights Reserved.
Outline
EDUCATION
Networking
Technologies

Outline
• Network vs. Storage • Layered ProtocolsEDUCATION
• ANs and ASes • Flow Control
• Perspectives • Payload Integrity
• Virtual SCSI Cables • Names vs. Addresses
• Port vs. PHY • Segmentation,
• Switch vs. Router vs. Convergence,
Gateway Reassembly & Offload
• Circuit Switch vs. Packet • Bit Rate, Bandwidth,
Switch Throughput
• Connection-Oriented vs. • Semantic Analysm
Connectionless • Future & Conclusions

Network Vs. Storage
EDUCATION
Network Considerations Storage Considerations

Information movement Information repository
Data over Distance Data over Time
Configuring and segmenting Formatting and partitioning hard
network topologies disks, Creating logical volumes
Hubs, Bridges, Switches, Routers, JBOD, RAID, Disk Controllers, Tape
Gateways Controllers
Client / Server Applications Initiator / Target Functions
Email, Web Browsing, File Mgt, Backup, Archive, Mirror, Block Mgt,
Network Interface I/O Interfaces

Network Vs. Storage
EDUCATION
Network Considerations Storage Considerations

Speed, Solid-State Capacity, Mechanical Movement
Media – Copper, Fiber, Wireless Media – Disk, Tape, Paper
Bits in Frames, CRC error checks Bytes in Blocks, Parity check every
every frame byte
Buffers and Link transfer rates – Cache and Disk/Tape I/O interface
Kbps, Mbps, Gbps transfer rates - MBps
Interconnectivity network and Disk/Tape seek times and latencies
device latencies - bottlenecks – ms, us, ns
Applications – HTTP, NFS, SMTP Logical Device Protocols – SCSI,
Transport Protocols – ESCON, IDE/ATA
TCP/IP/Ethernet Physical Protocols – Parallel SCSI,
Fibre Channel,
iSCSI/TCP/IP/Ethernet

Storage Data Units of Measure
Really Big Numbers EDUCATION
Unit of Abbreviation Approximate

Measure Size How long would it
Byte Byte 8 bits take someone to
Block Blk 512 Bytes download a one-
Kilobyte KB 1,024 Bytes yottabyte file using
Megabyte MB 1 Million
a 56 Kbps modem?
Bytes
Gigabyte GB 1,000 MB
Terabyte TB 1,000 GB
Petabyte PB 1,000 TB
Exabyte EB 1 Million TB
Zetabyte ZB 1,000 EB 5.5 trillion years.
Yottabyte YB 1,000 ZB

Network Data Units of Measure
Really Small Numbers EDUCATION
Unit of Power Number

Measure of 10
milli -3 One thousandth
micro -6 One millionth
nano
pico
-9
- 12
One billionth
One trillionth
1 1
femto - 15 One quadrillionth
atto - 18 One quintillionth
zepto - 21 One sextillionth
yocto - 24 One septillionth

Basic Storage Network Technology
EDUCATION
Direct-Attached
Storage (DAS)
Application
File System
Storage

EDUCATION
Direct-Attached Direct-Attached Direct-Attached

Storage (DAS) Storage (DAS) Storage (DAS)
Application Application Application
File System File System File System
Storage Storage Storage

EDUCATION
Direct-Attached Network-Attached
Storage (DAS) Storage (NAS)
Application Application
Network
File System File System
Storage Storage

EDUCATION
Network-Attached
Storage (NAS)
Network
File System
Storage

EDUCATION
Direct-Attached Network-Attached Storage Area

Storage (DAS) Storage (NAS) Network (SAN)
Network
File System
File System Network

File System

EDUCATION
Storage Area
Network (SAN)
File System File System File System
Network

EDUCATION
Direct-Attached Network-Attached Storage Area

Storage (DAS) Storage (NAS) Network (SAN)
Network File System

File System
File System Network

EDUCATION
Network-Attached Storage Area

Storage (NAS) Network (SAN)
Network File System
File System
Network
Storage Storage
16
EDUCATION
Network-Attached Storage Area

Storage (NAS) Network (SAN)
Network File System
NAS & SAN Network

File System
Storage Storage
17
DAS
EDUCATION
File File Request File Request

Application
Blocks
File System File
Block Request Block Request
Storage
Blocks

Network Attached Storage
NFS/CIFS Server EDUCATION
File
•Messages
NAS •Files
Blocks
NFS/CIFS
Client LAN NAS
File Request

Storage Area Network
EDUCATION
Storage
• Commands
• Blocks
NAS • Control
Gateway
• Status
SAN
SERVER

Storage Area Network – “External SAN”
EDUCATION
SCSI Target & LUN
Server Status
Blocks
SAN
SCSI Initiator
Client
Read
SERVER
File
Command
Request
“External SAN”
File Request
Application

LAN MAN WAN
EDUCATION
Network
Network
Network
Controller
A
“SmAN” Controller
Network Disk
Drives
B
Redundant Array of Independent Disks

LAN/SAN Benefits
EDUCATION
• Greater Asset
Utilization
• Expanded connectivity LAN
• Port consolidation
• Extended distance
FC IP
• Higher bandwidth
• Potential performance
improvement SAN

LAN & SAN Information Flow Requirements
EDUCATION
LAN SAN
iSCSI, HTTP,
SMTP, NFS,
FTP, DNS
TCP, UDP
IP
Ethernet

“External SAN” I/O Request to Storage Controller
EDUCATION
Disk
Disk
Controller
SAN Disk
Disk
Disk
Controller
Disk
Blocks Blocks
Status
External Read
Command
Request External SCSI
Target & LUN
“Internal SAN” I/O Requests in Storage Controller
Status
Internal SCSI
EDUCATION
Target & LUN (Disk
Disk Drive)
Disk
Blocks
FC
Controller 2
SAN Loop
Switch Disk Status
Blocks
Internal SCSI
Target & LUN
Disk (Disk Drive)
Disk
Controller 1
Internal Read Controller 1 Disk
Command Controller 1
Request Internal SCSI
Initiator Client
Status Controller 1 “Internal SAN”
Blocks Blocks Internal Read
Command RAID
External Read Request
Command
External SCSI
Request
Target & LUN
Status
Internal SCSI
EDUCATION
Target & LUN (Disk
Disk Drive)
Disk
Blocks
FC
Loop
Controller 2 Switch Disk Status
Blocks
Internal SCSI
Target & LUN
Disk (Disk Drive)
Disk
Controller 1
Internal Read Controller 1 Disk
Command Internal SCSI
Request Initiator Client
Controller 1
Status
Internal Read “Internal SAN”
Blocks Blocks Command
External Read
Request RAID
Command
External SCSI
Request
Target & LUN
Status
Internal SCSI
EDUCATION
Target & LUN (Disk
Disk Drive)
Disk
Blocks
SAS
Expander
Controller
Disk Status
Blocks
Internal SCSI
Target & LUN
Disk (Disk Drive)
Disk
Internal Read
Command Controller Disk
Request Internal SCSI
Initiator Client
Internal Read
Status
Command “Internal SAN”
Blocks Blocks Request
External Read RAID
Command
External SCSI
Request
Target & LUN
Virtual SCSI Cables on SCSI Bus
EDUCATION
Initiator Target
Host
Computer Host SCSI Bus Peripheral
Adapter Adapter Controller
P-SCSI
Initiator
0 1 2 62 63
Host Target
Host Logical Units
Computer Adapter
Adapter
Peripheral
Initiator Controller
Host
Computer Host 0 1 2 62 63
Adapter
Logical Units

Virtual SCSI Cables on SCSI Bus
EDUCATION
Initiator Target
Host
Computer Host SCSI Bus Peripheral
Initiator
0 1 2 62 63
Host Target Blocks
Host Logical Units
Computer Adapter
Adapter
Peripheral
Host
Adapter
Logical Units
SCSI Bus: Arbitrate ID, Select ID, Identify LUN

Virtual SCSI Cables Fibre Channel
EDUCATION
Initiator
Host Target
Computer Host Peripheral
Initiator FC SAN 0 1 2 62 63
Host Target
Host Logical Units
Computer Adapter
Adapter
Peripheral
Host
Adapter
Logical Units
FC: Fabric Login, N_Port Login, FC-4 Login

EDUCATION
Initiator
Host FC Switch
Computer Host Target
Adapter Peripheral
Adapter Controller
Initiator
Host FC SAN 0 1 2 62 63
Computer Host
Adapter Logical Units
Target
Initiator Adapter
Host Peripheral
Computer Host Controller
Adapter
0 1 2 62 63
Logical Units

EDUCATION
Initiator
Host FC Switch
Adapter Peripheral
Adapter Controller
Initiator
Host 0 1 2 62 63
Computer Host Blocks
Adapter Logical Units

Target
Initiator Adapter
Host Peripheral
Computer Host Controller
Adapter
0 1 2 62 63
Logical Units
FC Loop: Arbitrate, FC Fabric: Switch FC Point-to-Point:
Open Loop Circuit Routing Direct Routing
Virtual SCSI Cables SAS
EDUCATION
Initiator
Host
Computer Host
Adapter Peripheral
Adapter Target
Controller
Initiator
Host SAS 0 1 2 62 63
Computer Host
Logical Units
Adapter
Initiator
Adapter
Host Peripheral
Controller
Adapter
0 1 2 62 63
Logical Units

Virtual SCSI Cables SAS
EDUCATION
Initiator
Host SAS Expander
Computer Host
Adapter Peripheral
Adapter Target
Controller
Initiator
Host SAS 0 1 2 62 63
Computer Host
Logical Units
Blocks
Adapter
Initiator
Adapter
Host Peripheral
Controller
Adapter
0 1 2 62 63
Logical Units
Serial Attached SCSI: Open, Information, Close

Virtual SCSI Cables IP Storage (iSCSI)
EDUCATION
Initiator
Host
Computer Host
Adapter Peripheral
Adapter Target
Controller
Initiator
Host IP Storage 0 1 2 62 63
Computer Host
Logical Units
Adapter
Initiator
Adapter
Host Peripheral
Controller
Adapter
0 1 2 62 63
Logical Units

Virtual SCSI Cables IP Storage (iSCSI)
EDUCATION
Initiator
Host Ethernet Switch
Computer Host
Adapter Peripheral
Adapter Target
Controller
Initiator
Host 0 1 2 62 63
Computer Host
Logical Units
Blocks
Adapter
Initiator
Adapter
Host Peripheral
Controller
Adapter
0 1 2 62 63
Logical Units

M C Escher – Ascending & Descending Monks
EDUCATION
38
SAS Storage Controller Perspective
EDUCATION
Host Edge Fanout

HBA Expanders Expander
Disks Root
SAS “Expanse”
Service Delivery Subsytem

SAS Storage Network Perspective
EDUCATION
Fanout
Edge Expander
Expanders
HBA Disk
Root
End Devices Expanders
SAS “Expanse”
Service Delivery Subsytem

SAS Storage Network: Host – Disk Perspective
EDUCATION
Edge
Expanders
HBAs Disks
Fanout
Expander
Roots

SAS Storage Network: Host – Disk Perspective
EDUCATION

Parts of a Physical Transport Network?
EDUCATION
1. 2. 3.
Media or the Ports - 2 or more Protocol
“appearance of”
media
(transceivers)
?

Port vs. PHY = Ports vs. Docks
EDUCATION
Fibre Channel, Ethernet

1 Port – 1 PHY
SAS, Infiniband
1 Port – Many PHYs

Striping the Transmission Word
1 Link - 4 Phy Lanes
EDUCATION
0 1 2 3
0 1 2 3
PCIe, InfiniBand does Striping (4x, 12x)

SAS is not Striping – See connection path

Striping the Transmission Word
1 Link - 4 Phy Lanes
EDUCATION
0 1 2 3
0 1 2 3
10 Gbps Fibre Channel, PCIe, InfiniBand does Striping

SAS is not Striping – See connection path

SAS Expander – Open, Arbitration, Open
Accept
EDUCATION
ID OPEN 02, 03 ID
01 02
OPEN Accept
Expander
Node Node
ARB 02, 03
AIP Done
AIP
ID ID
OPEN 02, 03
03 04
Node Node
Frame Primitive

SAS Expander – Move
EDUCATION
ID ID
01 02
Expander
Node Node
ID ID
03 04
Node Node
Frame Primitive

SAS Expander – Move
EDUCATION
ID ID
01 02
Expander
Node Node
ID ID
03 04
Node Node
Frame Primitive

What is a Physical Transport Network?
• A group of connected systems - at least two! EDUCATION
– Host, Node, Computer, Server, Workstation, Printer, Router, Storage Device

Network Interface Cards (NICs), Host Bus Adapters
(HBAs), Storage Adapters (SAs)
• A media or the “appearance of” media
– Unshielded Twisted Pair(s) (UTP), Coaxial Cable(s), Multi-Mode Fiber(s),
Leased Line(s), RF
• A connection protocol or method
– Ethernet, Token Ring, PPP, Frame Relay, ATM, Fibre Channel, SONET/SDH,
DWDM
• Switches create an appearance of media
File
Server
Router
Switch

Routing in Switches
• Routing EDUCATION
• Switch frame “steering” over one

“appearance of media”

Routing in Routers
• Routing EDUCATION
• Router packet “forwarding”

across many “appearances of
media”
– A Day in the Life at Manhattan starts at
home:
– Walk to the bus stop
– Get on the bus
– Take the bus to the dock
– Get off the bus and walk on to the Ferry boat
– Sail across the river
– Get off the boat and hop in a cab.
– Drive to the office and get out of the cab.
– Run to the elevator and hop in
– Jump off the elevator and walk to your desk
– How many subnets? 9

An Internetwork: Switch vs. Router
• Routers join appearances of media EDUCATION
File
Server File
Server
Router
Switch
Switch
File
Server File
Server
Router
Switch
Switch

Router vs. Gateway
EDUCATION
File
Server File
Server
Router
Switch
Switch
Gateway
File
Server File
Server
Router
Switch
Switch

Circuit Switch – Packet Switch
EDUCATION
• Fixed path whether moving or not versus routing

frames or packets on the current best path as they
arrive and leave

Circuit Switch
• Circuit Construction
EDUCATION
• SAS Expanders Frame

• FC Class 1 Frame
• POTS (Plain Old
Telephone Network)

Packet (Frame) Switch
• Frame Direction
EDUCATION
• FC Class 3
Frame (switch)
• IP Packet (router)
• VOIP

Connection-Oriented vs. Connectionless
EDUCATION
• Not path related!

• Connection-oriented requires a state exchange
to track communications
• Connectionless takes things as they come
Telephone Telegraph
Connection-Oriented vs. Connectionless
EDUCATION
TCP
IP
TCP
IP
IP
IP
• Internet Protocol Suite

– TCP connections start with negotiation setup between hosts
• Similar to telephone connection
– Call, Answer, Talk, Hang up
• UDP connectionless
– IP Connectionless – Handles packets as they come
• Fibre Channel Class 2, 3 Switch Routing
– Connectionless – Handles frames as they come

Connection-Oriented FC Sessions - Login
EDUCATION
Server Storage
Link Link Link
Active Active Active
Fabric
Fabric Fabric
Login Login
N-Port
Login
FC-4
Process
Login
Layered Protocols
EDUCATION
• Protocol Data Units (PDUs)
• They are both cake – Different ways to

provide similar function!

OSI & IPS (TCP/IP)
EDUCATION
Open Systems Interconnection Internet Protocol Suite

End User Data End User Data
Application
Application
Presentation
Services
Session
Transport
Transport
Network Internet
Data Link Network

Interface
Physical Sublayer

OSI & Fibre Channel Architecture
EDUCATION
Application
Channels Networks
Presentation
IPI, SCSI, 802.2 (LE),
HIPPI, SBCCS IP, ATM
Session
Transport FC4 - Upper Layer Mapping
Network FC3 - Common Services
Data Link FC2 - Framing/Flow Control

FC1 - Encode/Decode
Physical FC0 - Physical

OSI & Fibre Channel Architecture
EDUCATION
Application
Channels Networks
Presentation
Session
Transport FC4 - Upper Layer Mapping
Network FC3 - Common Services
Data Link FC2 - Framing/Flow Control

FC1 - Encode/Decode
Physical FC0 - Physical

Fibre Channel & IPS
EDUCATION
Fibre Channel Internet Protocol Suite
Channels Networks
Application
Services
FC4 - Upper Layer Mapping Transport
FC3 - Common Services

Internet
FC2- -Framing/Flow
FC2 Framing/FlowControl
Control
Network
FC1 - Encode/Decode Interface
FC0 - Physical Sublayer

Flow Control
EDUCATION
• Fibre Channel
– Credit-Based Fibre Channel
– Advertise Credit
• Internet Protocol Suite

– TCP Sliding Window
– IP Datagram Drop Congestion Control
– Advertise Larger Windows

SAS Flow Control-
EDUCATION
Expanders – Circuit Switches
HBA Switch Switch SA
Frame = ~1K Max
Credit R-RDY = 1 Dword

Fibre Channel Buffer-to-Buffer (Class 2, 3)
EDUCATION
N-Port F-Port E-Port E-Port F-Port N-Port
HBA Switch Switch SA
B-B B-B B-B
Frame = ~2K Max
Credit R-RDY = 1 Word

Fibre Channel End-to-End (Class 1, 2)
EDUCATION
E-Ports
N-Port F-Port F-Port N-Port
Node Node Node Node
Fabric
Frame E-E
ACK Frame

Internet - TCP Sliding Windows
EDUCATION
• Sliding Windows
– Reduces overhead and delay of positive acknowledgments
– Allows sending of multiple segments before ACK is required
• Window Advertisements
– Receivers advertise window sizes
– Usually MSS multiple
– Maximum 65,535 Bytes
– Prevent “Silly window” syndrome
• Slow Start
– Congestion Window
– Increments till congestion detected

TCP Window Scale Factor
EDUCATION
• Multiplier for window field
• Window field only 16 bits for maximum of 65,535
bytes
• Both systems in a connection must agree to use
this
• Allows a maximum of 16 million bytes

TCP Sliding Windows – TCP Segments
EDUCATION
01 02 03 04 05 06 07 08 09 10
Send 6 Segments without
Acknowledgement
2 Acknowledgements
01 02 03 04 05 06 07 08 09 10
Send 2 More Segments

TCP Sliding Windows
Increasing EDUCATION
11 12 13 14 15 16 17 18 19 20
Decreasing
11 12 13 14 15 16 17 18 19 20

Payload Integrity
EDUCATION
• Do you always want

payload integrity?
• Not necessarily with
Video and Voice
• Performance vs. Quality
tradeoffs
– A pixel in a online-rented
movie frame
– A medical imaging application
just prior to heart surgery
where you want to be sure the
speck of grey isn’t line noise!

Names & Addresses
EDUCATION
Name
Address
Route

Name, Address & Route
EDUCATION
• Addressing and Identification schemes

Name
• Address is used to locate Address
• Names or identifiers uniquely represent

something or someone

Name, Address & Route
• Name is a specific identification EDUCATION
– Machine, User, Application - Unique target for data

• Internet
– Fully Qualified Domain Names WWW.SNIA.ORG
– Media Access Control MAC ID – Physical Port Identifier
• Storage
– World Wide Node Names WWNN (Fibre Channel)
– World Wide Port Names WWPN (Fibre Channel)
– Enterprise Unique Identifier EUI (iSCSI)
– iSCSI Qualified Name IQN (iSCSI)
• Address identifies where the target is located
– Hardware, Network, Port, Socket
– Port ID FCID (Fibre Channel)
– IP Address
• Route identifies path to get to the destination

Transport Addressing Ports
EDUCATION
• Ports are used to deliver information to relevant
application services
• Well-known and “Not-so-well-known” Transient
Port Name Description
20 FTP-3DATA File Transfer (Data Channel)
21 FTP File Transfer (Control Channel)
22 TELNET Telnet
25 SMTP Simple Mail Transfer
53 DNS
DOMAIN DomainDomain
NameName
Services
Service
3260 ISCSI Internet SCSI
• Comparable to Well known FCID Port ID Addresses

– FFFFFC Name Server
– FFFFFD Fabric Controller
– FFFFFE Fabric Port Login Server

Comparative Address Ranges
• 1 Address = 1 Gram EDUCATION
• SATA Point-to-Point
– 20 = 1
• SATA II “Port Multiplier”
– 24 = 16
• SAS “Expanders”
– 214 = 128 x 128 = 16,384
• All Fibre Channel
– 224 = 16,777,216
• All IPv4
– 232 = 4,294,967,296 X 56
Billion
• All IPv6
– 2128 = 340,282,366,920,938,463,463,374,607,431,768,211,456

Hierarchical, Routable Addressing
EDUCATION
• IP addresses can be divided into three parts
– Network Address (NETID)
– Subnetwork Address using (Subnetwork Mask 255.255.255.0)
– Host IP Address
• Example 172.16.2.6
– Network 172.16.0.0, Subnetwork 172.16.2.0, Host IP Address
172.16.2.6
• FC addresses can be divided into three parts

– Domain (Domain ID)
– Area (Area ID)
– Device (Device ID)
• Example 0102EF
– Domain 01 (Switch 01), Area 02 (Port 02 on the Switch), Device EF
(ALPA of Loop Device)

Address Assignment
• DHCP Mechanism for allocating and assigning IPEDUCATION
addresses to clients
– Automatic: Host requesting an address are provided with a
permanent IP address
– Dynamic: Hosts requesting an address are provided with a
temporary address
– Manual: Host IP addresses are manually configured and DHCP
just delivers these assignments
• Fibre Channel
– Loop Initialization LISM, LIFA, LIPA, LIHA, LISA, LIRP, LILP,
Close
• Select Master, Fabric Assigned, Previously
Assigned, Hard Assigned, Soft Assigned, Report
Position, List Position, Close
– Fabric Login Controller
– Point-to-Point - Lowest World Wide Port Name (WWPN) Assigns

Name Service
EDUCATION
• Internet
– Domain Name Services for IP
• Fully Qualified Domain Name FQDN to IP Address
• Running on distributed servers
• Storage
– Name Services for Fibre Channel
• WWPN to Fibre Channel Identifier FCID, Fabric
service running in the Switch
– iSNS Storage Name Service (iSNS) for iSCSI

Segmentation & Offload
EDUCATION
• Segmentation, Convergence and

Reassembly
• Offload for Performance Acceleration

Traditional FC SAN
Host NIC
EDUCATION
Internet
Protocol Suite Network
Host Interface
Host Sublayer
Applications
Applications
Application
Ethernet
FileManager
File Manager Services
DeviceDriver
Device Driver Transport
HBA
SCSIProtocol
SCSI Protocol Internet FC4
FC3
Operating System
FC2
I/O Interface FC2
FC1
FC0

Read 8K (6 FC frames with 4 - 2K Frames)
EDUCATION
Host
HBA Read Command

FC4 Data
FC3 Data
Data
FC2
FC2 Data
FC1
FC0
Status
Fibre Channel Multi-Frame Sequence Minimizes Server I/O interrupts - 2

Traditional NAS
Host NIC
EDUCATION
Internet
Protocol Suite Network
Host Interface
Host Sublayer
Applications
Applications NFS/CIFS
Application
Ethernet
FileManager
File Manager Services
DeviceDriver
Device Driver Transport
HBA
SCSIProtocol
SCSI Protocol Internet FC4
FC3
Operating System
FC2
FC2
I/O Interface
FC1
FC0

Get 8K (7 Ethernet Frames with 6 - 1.5K Frames)
EDUCATION
Host
NFS / CIFS / iSCSI
Application
Services NIC Get File
Data
Transport Network Data
Interface
Sublayer Data
Data
Internet
Ethernet Data
Data
Host IP Fragmentation - 6 IP Packet Fragments, 6 – 1500 byte

MTU Ethernet Frames, Maximizes Server I/O interrupts - 7

iSCSI Full Offload
Host EDUCATION
Intelligent
Internet NIC/HBA
Protocol Suite
Host
Host iSCSI
Applications
Applications i
S Application TCP
FileManager
File Manager C Services
S IP
DeviceDriver
Device Driver I
Transport
Network
SCSIProtocol
SCSI Protocol Internet Interface
Sublayer
Operating System Gigabit
I/O Interface Ethernet

Read 8K (8 Ethernet frames with 6 – 1.5K)
EDUCATION
Intelligent
NIC/HBA
Host Offload IP Fragmentation
iSCSI (Like FC Multi-Frame Sequence)
Minimizes Server I/O interrupts - 2
TCP
Read Command
IP
Data
Data
Network Data
Interface
Sublayer Data
Data
Gigabit
Ethernet Data
Status

Serial Scaling
EDUCATION

Serial Scaling
EDUCATION

Serial Link Bit Rate Scaling
1 Bit @ 1.5 Gbps = .666ns
EDUCATION
Assume 40 Bits = Tword
40 Folds
1 Bit
Transmit Receive
1 Bit
40 Bits = Tword
1 Bit @ 3.0 Gbps = .333 ns

Bandwidth
EDUCATION

iSCSI Overhead
iSCSI/ TCP/IP/ Ethernet
Ethernet with Non- EDUCATION
Overhead with Non-
Standard Standard Jumbo iSCSI/TCP/IP/ Standard Standard Ethernet Jumbo
iSCSI Ethernet Frames Ethernet Overhead Frames
iSCSI Data Out or Data in PDU

Header Bytes 52 52
TCP Header Bytes 20 20
IPv4 Header Bytes 20 20
Preamble Bytes 7 7 7 7
SFD Bytes 1 1 1 1
Destination Address Bytes 6 6 6 6

Source Address Bytes 6 6 6 6
Length/Type Bytes 2 2 2 2
Maximum Data/Pad Bytes 1,500 9,000 1,500 9,000
FCS Bytes 4 4 4 4
Ethernet Frame overhead

including Preamble and SFD
fields Bytes 26 26 26 26
Total Overhead Bytes 52 52 144 144

iSCSI Performance
EDUCATION
iSCSI/ TCP/IP/ Ethernet

Ethernet with Non- Overhead with Non-
Standard Standard Jumbo iSCSI/TCP/IP/ Standard Standard Ethernet
iSCSI Ethernet Frames Ethernet Overhead Jumbo Frames
Total Overhead Bytes 52 52 144 144
Payload Bytes 1,500 9,000 1,500 9,000
Efficiency % 98.30% 99.43% 91.24% 98.43%
Bit Rate gbps 1.25 1.25 1.25 1.25
Data Rate MBps 122.87 124.28 114.05 123.03

Fibre Channel Performance
EDUCATION
Bytes Bytes Bytes

SCSI FC SCSI FC
DATA Out or DATA Out or
Fibre IN / Fibre IN / Fibre
Channel Channel Channel Class
Fibre Channel Class 3 Class 3 2
SOF Bytes 4 4 4 4 4 4 4 4 4
Frame Header Bytes 24 24 24 24 24 24 24 24 24
Maximim Payload Bytes 2,112 2,048 2,048 2,112 2,048 2,048 2,112 2,048 2,048
CRC Bytes 4 4 4 4 4 4 4 4 4
EOF Bytes 4 4 4 4 4 4 4 4 4
InterFrame Word
Minimum Bytes Bytes 24 24 24 24 24 24 24 24 24
Acknowledgement
Frame Overhead 60 60 60
Total Overhead Bytes 60 60 120 60 60 120 60 60 120
Payload Bytes 2,112 2,048 2,048 2,112 2,048 2,048 2,112 2,048 2,048
Efficiency % 97.24% 97.15% 91.92% 97.24% 97.15% 91.92% 97.24% 97.15% 91.92%
Bit Rate gbps 1.0625 1.0625 1.0625 2.125 2.125 2.125 4.25 4.25 4.25
Data Rate MBps 103.31 103.23 97.67 206.63 206.45 195.33 413.26 412.90 390.66

SAS Performance
Bytes Bytes
EDUCATION
Serial Attached SCSI SAS SAS
SOF Bytes 4 4
Frame Header Bytes 24 24
Maximum Payload Bytes 1,024 1,024
CRC Bytes 4 4
EOF Bytes 4 4
InterFrame Word Minimum Bytes (1 Align every

2048 dwords) Bytes 2 2
R-Rdy Frame Overhead Bytes 4 4
Acknowledgement Frame Overhead Bytes 4 4

Total Overhead Bytes 46 46
Payload Bytes 1,024 1,024
Efficiency % 95.70% 95.70%
Bit Rate gbps 1.5 3
Data Rate MBps 143.55 287.10

Bandwidth does not equal bit rate
SAN Performance Comparison EDUCATION
Bit Rate Efficiency Data Rate Full Duplex

Standard Gigabit 1.25 Gbps 98.3% 123 MBps 246 MBps
Ethernet
iSCSI / TCP / IP / 1.25 Gbps 91.2% 114 MBps 228 MBps
Gigabit Ethernet
Standard MTU
1508
iSCSI / TCP / IP / 1.25 Gbps 99.43% 124.28 MBps 248.56 MBps
Gigabit Ethernet
Jumbo Frames
MTU 9000
Standard SCSI 1.0625 Gbps 97.2% 103.5 MBps 207 MBps
FCP Fibre
Channel Class 3
SAS 1.5 Gbps 95.7% 143.55 MBps 287.10 MBps

Harry Newton’s Telecom Dictionary 17 th
edition
• Delay EDUCATION
– The wait time between two events, such as the time from when a
signal is sent to the time it is received.
– There are all sorts of reasons for delays
• Propagation Delay
• Queuing Delay
• Processing Delay
• Rotational Delay
• Satellite Delay
• Latency
– A fancy term for waiting delay. The time it takes to get
information through a network.
– Accumulated delay

Delay & Latency
• Delay EDUCATION
– A metric that measures how long

something is not happening
– (A packet waiting in a queue)
• Latency
– A metric that measures how long
it is between things happening
– (The time between when a
packet arrives at a device and
leaves a device
• Throughput
– Delay and Latency affect
throughput

Throughput does not equal Bandwidth
EDUCATION

Network Components
EDUCATION
• Network Topologies
– Data Networks
• Point-to-Point, Bus, Token Ring, Fabric
– Storage Networks
• Bus, Arbitrated Loop, Point-to-Point, Fabric
– Hybrids
• Network Types
– Local Area, Metropolitan Area, Wide Area, Storage Area
• Backbone Networks
• Network Interconnectivity
– Repeaters, Hubs, Bridges, Switches, Routers, Gateways

Storage Network Choices
Integrated Device Electronics (IDE) EDUCATION
Small Computer Systems Interface (SCSI) Bus “Very Skinny
“Very Small
Enterprise System Connectivity (ESCON)
Serial ATA, Serial Attached SCSI
Fibre Channel Fibre Channel
Storage over IP
iSCSI / TCP / IP / GE
internet SCSI (iSCSI)
iSCSI / TCP / IP / GE Transmission Control Protocol (TCP) IP Storage
Internet Protocol (IP)
Gigabit Ethernet (GE)
internet Fibre Channel Protocol (iFCP)
Fibre Channel Internet Protocol (FCIP)
internet Storage Name Service (iSNS)
InfiniBand
Infiniband
SRP
iSCSI Extension over RDMA (ISER)

Semantic “Anal”yst: Beware of Semantics
EDUCATION
Network terminology Gateway

misuse is rampant!
• By the Storage
Network Industry
Router
• By Vendors
• By Media
• By Educators Switch
Confusing the
customer!
Hub

Beware of Semantics
EDUCATION
Gateway
GROUTER
Router
SWOUTER BROUTER
Switch
Hub

Beware of Semantics
EDUCATION
Smart Cluster
Gateway
Storage
Intelligent GROUTER
Virtual
Enterprise
Router
SWOUTER BROUTER
Director Switch Bridge
Repeater Hub Loop Switch

Plumbing is Beautiful
EDUCATION

Storage Networking Futures
EDUCATION
• Upper layer protocol changes
– Lazy and Hyper reads and writes
– System SAN awareness, Dynamic discovery
• Higher bandwidth rates for aggregation and
higher speed processors for servers, storage
and adapters
• Focus on PCI Express and Infiniband
• External SANs with Fibre Channel and
iSCSI/TCP/IP/GE
• Internal SANs with SAS and SATA
• NAS & SAN integration

Networking Technologies: Concepts in Internal
and External Networked Storage
EDUCATION
• Many
– Storage Network Solutions
– Storage Network Technologies
– Many Opportunities for Benefit
and Value
• Thanks for coming!

– Hope this helps!!
– Please fill out your evaluation
forms!!!
– Good luck with Storage Networks

Outline
EDUCATION
Networking
Technologies

Q&A / Feedback
EDUCATION
• Please send any questions or comments on this

presentation to the SNIA:
tracknetworking@snia.org
Many thanks to the following individuals

for their contributions to this tutorial.
SNIA Education Committee
Howard Goldstein – Tutorial Manager, Developer & Presenter
Greg Schulz – Tutorial Review

SW Worth – Tutorial Review

EDUCATION
HGAI Storage Networking

Web Links & Bibliography
For more info contact:

Howard.Goldstein@HGAI.Com
Networking Technologies: Concepts in Internal and External Networked Storage©

Copyright 2007 Howard Goldstein Associates, Inc. HGAI All Rights Reserved.
Web Links
EDUCATION
http://www.t10.org T10 (SCSI)

http://www.ncits.org/ The NCITS home page
http://www.symbios.com/t10.1/ The (former) T10.1 Task Group
(SSA) home page
http://www.t11.org/ The T11 Technical Committee
home page (Fibre Channel, HIPPI,
IPI, SBCON)
http://www.scsita.org The T13 Technical Committee
home page (ATA & ATAPI)
http://www.fibrechannel.com Fibre Channel Industry
Association (a trade association)

Web Links
EDUCATION
http://www.infostor.com Storage Industry information

http://www.snia.org Storage Networking Industry
Association (a trade
association)
http://www.1394ta.org/ IEEE 1394 Trade Association
http://www.infinibandta.org Infiniband Trade Association
http://www.ultra160-SCSI.com/ Information about Ultra160 SCSI
http://searchstorage.com Useful storage informaton
http://www.jobstor.com Jobs in the storage industry
http://www.asnp.org Association of Storage Network
Professionals
http://www.storagenetworking.org Storage Networking User
Groups

Bibliography
EDUCATION
• Fibre Channel – A Comprehensive Introduction Robert W. Kembel

• IP SANs– A Guide to iSCSI, iFCP, and FCIP Protocols for Storage Area
Networks Tom Clark
• Building Storage Networks Marc Farley
• iSCSI – The Universal Storage Connection John L. Hufferd
• Resilient Storage Networks – Designing Flexible Scalable Data
Infrastructures Greg Schulz
• Using SANs and NAS W. Curtis Preston
• Storage Area Network Essentials – A Complete Guide to
Understanding and Implementing SANs Richard Barker & Paul Massiglia
These texts were used in creating this course. Some of the course
material contains references from this material

Bibliography
EDUCATION
• Storage Security – Protecting SANs, NAS, and DAS John Chirillo &
Scott Blaul
• Inside Windows Storage – Server Storage Technologies for Windows
2000, Windows 2003, and Beyond Dilip C. Naik
• Successfully Implementing and Scaling SANs in the Enterprise - A
Planning Guide for Storage Networking Professionals
SearchStorage.Com
• The Holy Grail of Network Storage Management Jon William Toigo
• Storage Areas Networks for Dummies Christopher Poelker & Alex Nikitin
• Practical Storage Area Networking Daniel Pollack
• Designing Storage Area Networks – A Practical Reference for
Implementing Fibre Channel and IP SANs Tom Clark

Bibliography
EDUCATION
• Storage Area Networks – Designing and Implementing a Mass Storage

System Ralph H. Thornburgh & Barry J. Schoenborn
• Cisco Storage Networking Architecture Poster Cisco Press
• IP Storage Networking – Straight to the Core Gary Orenstein
• Fibre Channel for Mass Storage Ralph H. Thornburgh
• Making SCSI Work – A Practical Guide The Paralan Staff
• Fibre Channel for SANs Alan F. Benner
• The Book of SCSI – A Guide for Adventurers Peter M. Ridge
• The Holy Grail of Data Management – Modeling, Evaluating,
Comparing, Implementing, Storage Management Options, and
Solutions Jon William Toigo
• Storage Network Performance Analysis Huseyin Simitci

Bibliography
EDUCATION
• Fibre Channel – Gigabit Communications and I/O for Computer Networks Alan
F. Benner
• Building SANs with Brocade Fabric Switches – How to Design, Implement, and
Maintain Storage Area Networks (SANs) with Brocade Fabric Switches Chris
Beauchamp, Josh Judd, & Benjamin Kuo
• Multiprotocol Routing for SANs Josh Judd
• The Fibre Channel Bench Reference Jeffrey D. Stai
• The SCSI Bus & IDE Interface – Protocols, Applications & Programming
Friedhelm Schmidt
• Fibre Channel Switched Fabric Robert W. Kembel
• Fibre Channel Arbitrated Loop Robert W. Kembel
• Storage Networks Daniel Worden
These texts were used in creating this course. Some of the course material
contains references from this material

Bibliography
EDUCATION
• Disaster Recovery Planning – Preparing for the Unthinkable Jon

William Toigo
• Fibre Channel Volume 1: The Basics Gary R. Stephens & Jan V. Dedek
• Small Computer System Interface – An In-Depth Exploration Of ….
David Deming
• What is Fibre Channel? Ancot Corporation
• Fibre Channel Connection to the Future Fibre Channel Association FCA
• Introduction to Storage Area Network, SAN Ravi Kumar Khattar, Mark S.
Murphey, Giulio John Tarella, Kjell E. Nystrom

Bibliography
EDUCATION
• FireWire System Architecture Second Edition IEEE 1394a MindShare,

Inc. Don Anderson
• PCI-X System Architecture MindShare, Inc. Tom Shanley
• Storage Virtualization Technologies for Simplifying Data Storage and
Management Tom Clark
• Serial Attached SCSI: Beyond the Basics SAS Knowledge Tek Author:
Hugh Curley
• SAS Storage Architecture – Serial Atached SCSI MindShare, Inc. Mike
Jackson

Bibliography
EDUCATION
• Securing Storage A Practical Guide to SAN and NAS Security
Himanshu Dwivedi
• Storage Network Security – SNIA Technical Tutorial Roger Cummings &
Hugo Fruehauf
• Storage Virtualization – SNIA Technical Tutorial Frank Bunn, Nik
Simpson, Robert Peglar & Gene Nagle
• Storage Network Management – SNIA Technical Tutorial Roger
Cummings
• Network Storage Terms & Acronyms Storage Networking Industry
Association SNIA

Bibliography
EDUCATION
• Ethernet The Definitive Guide Charles E. Spurgeom

• Gigabit Ethernet for Metro Area Networks Paul Begell
• Gigabit Ethernet Networking David G. Cunningham, Ph.D. and William G.
Lane, Ph. D.
• High Performance TCP/IP Networking – Concepts, Issues, and
Solutions Mahbub Hassan, Raj Jain
• Gigabit Networks – Standards and Schemes for Next-Generation
Networking Paul Izzo


Networking Technologies Concepts Internal External Storage PDF

Transféré par

Informations du document

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Networking Technologies Concepts Internal External Storage PDF

Transféré par

Droits d'auteur :

Formats disponibles

EDUCATION

Networking Technologies: Concepts in Internal and External Networked Storage©

Networking Technologies: Concepts in Internal and External Networked Storage 2

Networking Technologies: Concepts in Internal and External Networked Storage 3

Networking Technologies: Concepts in Internal and External Networked Storage 4

Network Considerations Storage Considerations

Networking Technologies: Concepts in Internal and External Networked Storage 5

Network Considerations Storage Considerations

Networking Technologies: Concepts in Internal and External Networked Storage 6

Unit of Abbreviation Approximate

Networking Technologies: Concepts in Internal and External Networked Storage 7

Unit of Power Number

milli -3 One thousandth

micro -6 One millionth

femto - 15 One quadrillionth

atto - 18 One quintillionth

zepto - 21 One sextillionth

yocto - 24 One septillionth

Networking Technologies: Concepts in Internal and External Networked Storage 8

Networking Technologies: Concepts in Internal and External Networked Storage 9

Direct-Attached Direct-Attached Direct-Attached

Application Application Application

File System File System File System

Storage Storage Storage

Networking Technologies: Concepts in Internal and External Networked Storage 10

Networking Technologies: Concepts in Internal and External Networked Storage 11

Application Application Application

Networking Technologies: Concepts in Internal and External Networked Storage 12

Direct-Attached Network-Attached Storage Area

Application Application Application

File System Network

Storage Storage Storage

Networking Technologies: Concepts in Internal and External Networked Storage 13

Application Application Application

File System File System File System

Storage Storage Storage

Networking Technologies: Concepts in Internal and External Networked Storage 14

Direct-Attached Network-Attached Storage Area

Application Application Application

Network File System

File System Network

Storage Storage Storage

Networking Technologies: Concepts in Internal and External Networked Storage 15

Network-Attached Storage Area

Network File System

Network-Attached Storage Area

Network File System

NAS & SAN Network

File File Request File Request

Networking Technologies: Concepts in Internal and External Networked Storage 18

Networking Technologies: Concepts in Internal and External Networked Storage 19

Networking Technologies: Concepts in Internal and External Networked Storage 20

Networking Technologies: Concepts in Internal and External Networked Storage 21

Redundant Array of Independent Disks

Networking Technologies: Concepts in Internal and External Networked Storage 22

Networking Technologies: Concepts in Internal and External Networked Storage 23

Networking Technologies: Concepts in Internal and External Networked Storage 24

Networking Technologies: Concepts in Internal and External Networked Storage 29

SCSI Bus: Arbitrate ID, Select ID, Identify LUN

Networking Technologies: Concepts in Internal and External Networked Storage 30

FC: Fabric Login, N_Port Login, FC-4 Login

Networking Technologies: Concepts in Internal and External Networked Storage 31

Networking Technologies: Concepts in Internal and External Networked Storage 32