BRKDCN-1100 - MTU Considerations and Implications in The Data Center (2016 Las Vegas)

Size Matters: MTU Considerations
and Implications in the Data Center

Eric Spielman, Network Consulting Engineer
BRKDCN-1100
Agenda
Overview of MTU within the Data Center
Sizing across multiple protocols
MTU interaction with Data Center technologies
Ins & Outs of MTU related configuration
Summary
Session Goal
BRKDCN-1100
2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Overview of MTU within the

Data Center
The Data Center Evolution
VXLAN, OTV
FabricPath
vPC / vPC+
STP
Fabric
Extension
Unified Fabric
FCoE, iSCSI
STP Tiered Design
Full Link Utilization
Host & Network Overlays
Separate SAN Infrastructure
Converged Fabrics
Increased Scalability
BRKDCN-1100
Role of MTU
What is the maximum size of a

frame that a network device will
accept?
Bidirectional attribute
Hop-by-hop basis
BRKDCN-1100
Jumbo Frames
Any payload size greater than

1500 bytes
Does not include Ethernet headers

Enables 6x as much payload data in
one transmit
14 byte
Eth Hdr
20 byte
IP Hdr
20 byte
L4 Hdr
14 byte
Eth Hdr
20 byte
IP Hdr
20 byte
L4 Hdr
Data Payload
1460 bytes
4 byte
Eth FCS
Data Payload
9176 bytes
Nexus & UCS support payloads of

up to 9216 bytes
MTU 1500
4 byte
Eth FCS
MTU 9216
BRKDCN-1100
Why the Draw to Jumbo Frames?
Larger MTU is needed for protocols and technologies like FCoE, OTV,
and recommended for apps such as vMotion
Jumbo frames reduce the number of copies and context switches that
occur in the host due to packet overhead reduction
A 9000 byte frame results in up to 4 context switches and 4 copies to message

in, process, and exit the host
A typical 1500 byte frame would require 6x as many to receive the same
amount of data
Improved forwarding and buffer utilization from network devices
BRKDCN-1100
Finding the Right Size

PMTUD for IPv4
MTU 9000
R1
7K-1
Host
MTU 9000
MTU 1500
R2
1. Host sends 9000 byte packet to discover

max MTU
2. R2-R3 link is only 1500 MTU & needs to
fragment
3. DF-bit prevents & ICMP unreachable is
sent back to Host with link MTU of 1500
7K-2
Server
Transmit with new MTU value
BRKDCN-1100
10
PMTUD
In real life...
ISP
7K-1
7K-2
(config-if)# no ip unreachables
Host
Server
BRKDCN-1100
11
Effects of MTU Mismatches
Performance issues:
Lost packets
Retransmits
Truncation
High delay
Excess resource utilization
Issues with dynamic routing

adjacencies
Fragmentation...
BRKDCN-1100
12
The Issue(s) with Fragmentation...
Breaks down large IP datagrams into smaller datagrams so

that it can be sent on a link with a smaller MTU
Fragmentation introduces more router load

Host is responsible for reassembling = more load on the host
If one fragment is missing, the entire original packet must be
resent
Only initial fragment contains upper-level protocol headers
Non-initial fragments will match only source/destination IP

information in ACLs
May be vulnerable to attack
BRKDCN-1100
13
The Bigger The Better?

DC02
Jumbo frame support is not widespread
North / South and inter-DC traffic should be

cautious of going above 1500 MTU
Never assume
P1.a
P1.b
DC01
Doesnt play well with real-time traffic
More prone to performance issues in terms

of RTT and latency
P1
BRKDCN-1100
14
Its Good Where It Makes Sense
East / West traffic; the busy parts

of the network
Storage traffic
Workload mobility
Isolated traffic
Private WAN where there is control

& visibility into the complete traffic
path
All stars need to align
BRKDCN-1100
15
Sizing Across Multiple Protocols
Making It All Flow...
BRKDCN-1100
17
NX-OS System QoS Policies

Policy Application Point
Policy Type
Function
system
qos
ingress
interface
qos
Define traffic
classification
queuing
Apply queuing
strategy; strict
priority, deficit
weight round
robing
network-qos
Define system
class: drop/nodrop, MTU,
buffer size
egress
interface
Match and define policy using the respective type of class-maps and
policy-maps
BRKDCN-1100
18
QoS Policy Types

*Platform dependent criteria
type qos
type queuing
type network-qos
Class-map
Class-map
Class-map
CoS, DSCP, ACL, IPP,
CoS, Protocol
CoS, DSCP
Protocol (TLV)
qos-group
qos-group
Policy-map
Policy-map
Policy-map
Marking, policing, mutation
MTU, No-drop (lossless Ethernet)
Set qos-group
Congestion control
Priority, queueing, buffering,

scheduling
BRKDCN-1100
19
Type Network-QoS Policies
One per system that is applied to the

Unified Crossbar and affects all L2
ports
Controls the shape of the traffic and

how it reacts
Exception is native FC interfaces
Ability for varying MTU values among

classes
Why use CoS?

Ethernet Wire
BRKDCN-1100
20
IPv4 Traffic
MTU may be determined based on:

Interface configuration <mtu x>
network-qos policy: mapped into classes based on CoS/QoS marking or qos-groups
Otherwise the default class value is used
L3 hop where packet exceeds MTU: fragment except when the DF-bit is present
Switched traffic exceeding the MTU is truncated or dropped

Consistent MTU scheme
Ingress
Module
Ingress
Module
Ingress
Module
Fabric
Egress
Module
Ingress
Module
Ingress
Module
Ingress
Module
Fabric
Egress
Module
BRKDCN-1100
Ingress
Module
Ingress
Module
Ingress
Module
Fabric
Egress
Module
21
IPv6
No fragmentation supported on routers!
Must rely on PMTUD, consistent MTU scheme, RA messages
Hosts are responsible for fragmentation

ICMPv6 Type2 messages must be allowed for PMTUD operation
MTU for IPv6 traffic may be set with the interface-level MTU command
or within a network-qos class
BRKDCN-1100
22
IPv6
NX-OS has one interface-level MTU command: mtu <size>
Influences IPv4 & IPv6 alike
Recommendation: have IPv6 routers send hosts the MTU that should be used
via Router Advertisements (ICMP packets)
Instructs the host to use an MTU of

1480 in packets it originates
N7706-A(config-if)# ipv6 nd mtu 1480

N7706-A(config-if)# sh ipv6 nd int e1/11
ICMPv6 ND Interfaces for VRF "default"
Ethernet3/1, Interface status: protocol-down/linkdown/admin-down
IPv6 address: 0dc3:0dc3:0000:0000:0218:baff:fed8:239d
ICMPv6 active timers:
Last Neighbor-Solicitation sent: never
Last Neighbor-Advertisement sent: never
Last Router-Advertisement sent:never
Next Router-Advertisement sent in: 0.000000
Router-Advertisement parameters:
Periodic interval: 200 to 600 seconds
Send "Managed Address Configuration" flag: false
Send "Other Stateful Configuration" flag: false
Send "Current Hop Limit" field: 64
Send "MTU" option value: 1480
BRKDCN-1100
RA
23
iSCSI
Block-level storage access over IP
Can co-exist with LAN infrastructure

TCP or PFC
iSCSI with Jumbo Frames
GOOD: Allows for entire SCSI data blocks in

packets = less buffer wait time and throughput is
consumed more efficiently
NOT SO GOOD: MTU mismatches severely hinder
performance with potential disconnects and data
corruption
BRKDCN-1100
24
iSCSI Best Practices

WAN
Intra-DC iSCSI
with existing LAN
Inter-DC iSCSI
Multiple L3 hops
Remote
iSCSI
LAN
iSCSI direct
attach to UCS FI
SAN
Use Jumbo Frames when there is control

over all transit devices
Best practice: isolate iSCSI traffic (i.e.

dedicated replication/backup VLANs), apply
separate CoS value
BRKDCN-1100
25
Sample VLAN-based iSCSI Policy

Reference 5672 Platform
IP ACL matches the well-known
iSCSI ports for the QoS policy
Sets the internal qos-group for

system class mapping and
applied in the VLAN database
Network-qos policy matches the

iSCSI traffic with the qos-group
and increases the MTU
ip access-list iSCSI-ACL
permit tcp 10.0.0.0/24 any eq 860
permit tcp 10.0.0.0/24 any eq 3260
!
class-map type qos match-all class-iSCSI
match access-group name iSCSI-ACL
!
policy-map type qos iSCSI-QOS-policy
class class-iSCSI
set qos-group 2
set cos 4
class class-default
!
vlan configuration 10
service-policy type qos input iSCSI-QOS-policy
!
class-map type network-qos iSCSI-nq
match qos-group 2
!
policy-map type network-qos jumbo-nq-policy
class type network-qos c-nq-iSCSI
mtu 9000
pause no-drop
class type network-qos c-nq-default
mtu 1500
!
system qos
service-policy type network-qos jumbo-nq-policy
BRKDCN-1100
26
NAS
File-level storage access over an IP

network
NFS or SMB
Storage can be shared simultaneously across
multiple computers
TCP or UDP based
May benefit from increased MTU but not

required
LAN
SAN
Mismatches result in latency and slow

performance
Similar to iSCSI principles, assign a consistent
CoS/QoS policy
BRKDCN-1100
27
Fibre Channel
Supported on Unified Port (UP) switches that include the Nexus 5500/5600/6000
series switches as well as the 2348UPQ FEX
Requires Jumbo Frames
Maximum receive size of 2148 bytes with headers

Maximum payload of 2112 bytes
MTU adjustments are hard-coded when interface is converted to fc and traffic is

identified/classified based on the physical interface it is received on
SOF
FC Header
4bytes
24bytes
Frame delimiters
SCSI Payload
Commands & Data
2112 bytes
CRC
EOF
4bytes 4bytes
Control information,
src/dst addresses
BRKDCN-1100
28
Fibre Channel over Ethernet (FCoE)
Still Fibre Channel
FC frame is encapsulated with an FCoE header and regular Ethernet headers

Classified based on its EtherType
CoS 3 is the standard FCoE QoS marking and will auto-associate with this class
ELP requires that rxbufsize match between peers, otherwise link fails
2180 Bytes
Ethernet
Header
12bytes
+ 4bytes
.1Q tag
FCoE
Header
FC Header
16bytes
24bytes
SCSI Payload
Commands & Data
2112 bytes
CRC
EOF
FCS
4bytes 4bytes 4bytes
FCoE identified
with Ethertype
0x8906
BRKDCN-1100
29
FCoE Among Platforms
N56128-A# sh policy-map system

Type network-qos policy-maps
============================
policy-map type network-qos jumbo+_fcoe-nq-policy
class type network-qos class-fcoe
pause no-drop
mtu 2158
class type network-qos class-default
mtu 9000
N7706-Admin-A# sh policy-map system

============================
policy-map type network-qos jumbo-nq-7e-4q8q-policy template 7e-4q8q
class type network-qos c-nq-7e-4q8q-drop
congestion-control tail-drop
mtu 9000
class type network-qos c-nq-7e-4q8q-ndrop-fcoe
pause
mtu 2112
Header length is internally added and

implemented in hardware with fixed sizes
2112 + 18 (eth_hdr&CRC) + 28
(FC_hdr&CRC) = 2158 bytes
N7K: Must use a dedicated Storage

VDC and a network-qos policy
BRKDCN-1100
30
Fibre Channel over IP (FCIP)

FC SAN
FC SAN
FCIP
end-point
FCIP
end-point
Long distance SAN extension between FCIP end-points
Available on MDS platforms
Leverages TCP for resilience and congestion control
BRKDCN-1100
31
FCIP Frame
Ethernet
Header
14
IP
Header
TCP
Header
TCP
Opts
FCIP
Header
S
O
F
EISL
Hdr
Opt
Hdr
20
20
12
28
0-16
FCIP Overhead: 98 Bytes

Includes 94 byte header
& 4 byte CRC
FC Frame
CRC
Maximum FC frame size:

2148 bytes
FCIP will segment and reassemble FC frames if MTU exceeds
Enabling Jumbo frames across the path will increase performance
DF-bit is set between FCIP end-points

BRKDCN-1100
32
FCIP MTU & Discovery
FCIP MTU
recommendation is
2300 bytes
Path MTU
Discovery is
enabled by default
FCIP end-points
Increase MTU on physical

interface; will also increase
on FCIP interface
Any MTU changes across

path will be discovered
within 3600 secs
MDS-A(config)# feature fcip

MDS-A(config)# int gi1/1
MDS-A(config-if)# ip address 10.1.1.1/24
MDS-A(config-if)# mtu 2300
MDS-A(config)# fcip profile 10
MDS-A(config)# ip address 10.1.1.1
MDS-A(config)# int fcip 10
MDS-A(config-if)# use-profile 10
MDS-A(config-if)# peer-info ipaddr10.1.1.2
MDS-A(config-if)# no shut
MDS-A(config-if)# show fcip profile 10
FCIP Profile 10
Internet Address is 10.1.1.1 (interface GigabitEthernet1/1)
Tunnels Using this Profile: fcip10
Listen Port is 3225
TCP parameters
SACK is enabled
PMTU discovery is enabled, reset timeout is 3600 sec
Keep alive is 60 sec
Minimum retransmission timeout is 200 ms
Maximum number of re-transmissions is 4
Send buffer size is 16384 KB
Maximum allowed bandwidth is 5000000 kbps
Minimum available bandwidth is 4000000 kbps
Configured round trip time is 1000 usec
Congestion window monitoring is enabled, burst size is 50 KB
Auto jitter detection is enabled
BRKDCN-1100
33
MTU Interaction with Data Center

Technologies
vPC
Overview
vPC Logical View
vPC Physical View
STP
7K-1
7K-1
7K-2
7K-1
SW1
7K-2
7K-2
SW1
SW1
All links forwarding: no blocked ports, increased bandwidth, load sharing
High availability & redundancy
BRKDCN-1100
35
vPC
MTU Areas of Interest
vPC Domain
Peer-Link
vPC Peer-Keepalive
Internally programmed for 9216+

padding
Extra padding is due to Cisco Data
Center Ethernet Protocol (CDCE)
vPC Peer-Link
vPC Member
Ports
MAC-in-MAC
EtherType 0x8903
Overhead auto-accounted for
vPC
vPC
vPC
Downstream device
Should have same MTU as vPC

Member Ports
BRKDCN-1100
36
vPC
Member Port MTU
Applies to switches where interface-level MTU config is used
MTU config must match on vPC port-channel member interfaces between

vPC peers
Physical interfaces auto-inherit
Mismatch causes forwarding interruption for that specific vPC (Type-1)

N7706-A# sh vpc consistency-parameters inter po 11
Legend:
Type 1 : vPC will be suspended in case of mismatch
Name
------------mode
Speed
Duplex
Port Mode
Native Vlan
MTU
Admin port mode
Type
---1
1
1
1
1
1
1
Local Value
---------------------active
10000 Mb/s
full
trunk
1
9000
trunk
BRKDCN-1100
Peer Value
----------------------active
10000 Mb/s
full
trunk
1
9000
trunk
37
vPC
Global Consistency Check
Applies to Nexus switches with network-qos policies
vPC still operates despite MTU mismatch within the network-qos policy
(Type-2)
*No reason for it not to match

Nexus56128P-A# sh vpc consistency-parameters global
qos-group 0 (default)
qos-group 1 (FCoE)
Legend:
Type 1 : vPC will be suspended in case of mismatch
Name
------------QoS
Type
---2
Network QoS (MTU)
Network Qos (Pause)
Local Value
---------------------([], [3], [], [], [],
[])
(9216, 2196, 0, 0, 0,
0)
(F, T, F, F, F, F)
BRKDCN-1100
Peer Value
----------------------([], [3], [], [], [],
[])
(9216, 2196, 0, 0, 0,
0)
(F, T, F, F, F, F)
38
Why Overlays
Encapsulated Traffic
Virtual Switch
Virtual Switch
Resilient Underlay
Overlay Virtual Network
High capacity interconnects
Scalable
Transit devices are agnostic
Flexible
Must be capable of handling

overhead
L2 or L3
BRKDCN-1100
39
FabricPath
Benefits
S100
S10
S20
S30
S40
S200
S300
S400
S500
FabricPath
S600
Maintains Layer 2 connectivity without the use of STP
Efficient while utilizing ECMP
Enables flexibility within the Data Center and is highly scalable
BRKDCN-1100
40
FabricPath
Control Plane Operation
Utilizes IS-IS to dictate the forwarding topology
Essentially L3 routing to the L2 bridged domain

Computes shortest path within the FabricPath domain
Supports the use of equal-cost paths
No L2 IS-IS configuration needed but is tunable
IS-IS LSPs have an MTU of 1492 bytes

Can adjust with lsp-mtu <size> command but no documented performance benefit
since it is strictly control traffic
Recommend to leave at default
BRKDCN-1100
41
FabricPath
Data Plane
All fields retained, new CRC

S20
S10
OutDA
OutSA
FPTag
DMAC
SMAC
S100
16 bytes total overhead

within FP Domain
DMAC
.1Q
S200
SMAC
.1Q
Etype
S30
Payload
S300
Etype
CRC
S400
Payload
S40
S500
S600
CRC
FP header removed as
frame returns to CE domain
Classical Ethernet Frame
Ingress into FP Domain
BRKDCN-1100
42
FabricPath
Data Plane
The 16 bytes of overhead is automatically accommodated by the ASICs when

the interface is converted to switchport mode fabricpath
Same EtherType (CDCE) as on vPC Peer-Link
MTU configurations and adjustments on the platform are reflected in the

FabricPath forwarding domain:
N7706-Spine-A# show fabricpath isis interface brief
Fabricpath IS-IS domain: default
Interface
Type Idx
State
Circuit
MTU Metric Priority Adjs/AdjsUp
-------------------------------------------------------------------------------port-channel10 P2P
1
Up/Ready
0x01/L1
9000 10
64
1/1
port-channel20 P2P
Up/Ready
0x01/L1
9000
10
64
1/1
port-channel30 P2P
Up/Ready
0x01/L1
9000
10
64
1/1
BRKDCN-1100
43
OTV
OTV
L3 DCI
Overview
OTV Edge
OTV Edge
OTV Edge
OTV Edge
OTV Inline
LAN Extension over any network
Core
Dark Fiber, MPLS, IP...

Multi-data center scalability with
ability for workload mobility
Network-based overlay with

stateless tunneling
Core
Core
Core
West DC
East DC
VLANs 100-109
VLANs 100-109
OTV Join
Interfaces
Core Transit
Overlay Interface
OTV
Edge
Core Device
OTV on a stick
Aggregation Device
Internal Interfaces
BRKDCN-1100
44
OTV
Packet Types
Packet Type
Purpose
IP Length
ISIS Hello;
Unicast
Establish OTV
Adjacency
1450 Bytes
ISIS Hello;
Multicast*
Establish OTV
Adjacency
1442 Bytes
OTV Header
Added to all OTV

Data traffic
Pkt_len+42 Bytes
*Recommended
OTV control
In order to forward a standard 1500 byte packet from a host application,

ensure transport supports at least a 1542 byte MTU end-to-end
Alternatively the host MTU could be decreased to accommodate overhead

BRKDCN-1100
45
OTV
Data Packet Format
MAC-in-IP
802.1Q tag is removed from original L2

header
VLAN ID copied to OTV Shim
802.1P bits copied to new IP Header and
OTV Shim
Fragmentation is not supported due to L2

presumption and hiding of L3 network
OTV
Overhead
DF bit is set on all OTV control and data

traffic between Edge devices
New L2 Header
14 Bytes
New IP Header
20 Bytes
OTV Shim
8 Bytes
Original L2 Header
14 Bytes
Original IP Packet
Payload
Original
Frame
No PMTUD either
L2 FCS
BRKDCN-1100
46
OTV
MTU 1542 bytes
Interface Config
7K-A-CORE(config-if)#
int po10
description OTV Internal Int
mtu 1542
int e1/1
description Core Uplink
mtu 1542
OTV Edge
Device
Core Device
Aggregation Device
7K-A-OTV(config-if)#
int po10
description Join Interface
mtu 1542
int overlay1
otv join-interface po10
OTV Internal Interface

OTV Join Interface
BRKDCN-1100
47
VXLAN
What Does it Accomplish?
Flexible host-based overlay with

mobility across data centers and
clouds
Segmentation and Scale
Underlay
Stretches the L2 segment ID field to 24

bits = up to 16 million L2 segments
Extends L2 segments over a L3
boundary VXLAN encapsulates L2
frames in an IP/UDP header
VXLAN VNI 100
Allows for all links forwarding via L3

ECMP
BRKDCN-1100
48
VXLAN
1500 byte packet
Overhead and Transport
14 byte inner-Ethernet header
8 byte VXLAN ID
8 byte UDP header
VXLAN Encapsulation
20 byte IP header
50 Bytes (possibly 54)
1550 byte minimum VXLAN MTU
MAC Header
IP Header
UDP Hdr
14 bytes (4 bytes optional)_
20 Bytes
8 Bytes
Outer
MAC
DA
Outer
MAC
SA
Transport MAC
addresses
Outer
802.1Q
Outer
IP
DA
Outer
IP
SA
Outer
UDP
VXLAN
ID
(8 bytes)
Inner
MAC
DA
Inner
MAC
SA
VM MAC
addresses
Transport
VLAN
(Optional)
IP addresses
of the VTEPs
UDP destination
port 4789 identifies
the packet as
VXLAN traffic;
source UDP is a
hash of inner frame
VNI Field
identifies the L2
segment
BRKDCN-1100
Ethernet
Payload
CRC
The original frame; at

default 1514 bytes or
1518 bytes
49
VXLAN
Operation
Virtual Tunnel End Points

(VTEP) perform VXLAN
encapsulation / de-capsulation
Add 50 bytes to transit links to

support MTU of 1550 bytes
IPv4 Transport
VTEP
Hypervisor
VTEP
Alternatively MTU could be

decreased by 50 bytes on
servers
VTEP
LAN segment for bridging

& IP interface known in the
transport network (usually
a loopback)
When 1000v is used as VTEP,

it can assist with MTU issues
BRKDCN-1100
50
Ins and Outs of MTU Related

Configurations
Logical Flow
Operating System
Compute
Switching
Storage
OS Kernel
Driver
Adapter
IOM
FI
1000v
Nexus
Switch
IP
MTU
BRKDCN-1100
52
Influencing Traffic Size

IOS vs. NX-OS
IOS
NX-OS
ip mtu
mtu
mtu
network-qos policy
ip tcp adjust-mss
system mtu
...
BRKDCN-1100
53
Nexus 7000 & 7700 Series
Store-and-forward switch
May use interface-level MTU

commands or network-qos policy
depending on interface
Features such as FCoE and FEX

require network-qos policies
BRKDCN-1100
54
Interface-Level MTU
Layer 2
Jumbo MTU is supported on the

switch by default but needs to be
enabled
Interface-level must use the

standard default MTU of 1500 or
match the value specified in
system jumbomtu command
N7706-A(config)# sh run all | i jumbomtu

system jumbomtu 9216
N7706-A(config)# sh int e1/1
Ethernet1/1 is up
admin state is up, Dedicated Interface
Hardware: 1000/10000 Ethernet, address:
Description: 5K-A 1/4
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
N7706-A(config)# int e1/1
N7706-A(config-if)# mtu 9216
N7706-A(config)# sh int e1/1
Ethernet1/1 is up
admin state is up, Dedicated Interface
Description: 5K-A 1/4
Applies for port-channels as well

Nexus7706-A(config)# int e1/1
Nexus7706-A(config-if)# mtu 9000
ERROR: Ethernet1/1: MTU on L2 interfaces can only be set to default or system-jumboMTU
BRKDCN-1100
55
N7K network-qos Policies
Alternative MTU configuration is via the

network-qos policy in the Admin VDC
Applies to L2 traffic in all VDCs

Class-maps match on CoS and protocol
Disruptive
Default policy cannot be modified; copy

& modify a predefined template or
create one that conforms to a template
Any interface-level MTU will take

precedence
Example policy with 7 drop

classes & 1 no-drop (FCoE)
N7706-A# qos copy policy-map type network-qos default-nq-7e-4q8q-policy
prefix CUSTOMN7706-A# conf
N7706-A(config)#policy-map type network-qos CUSTOM-nq-7e-4q8q-policy
N7706-A(config-pmap-nqos)# class type network-qos c-nq-7e-4q8q-drop
N7706A(config-pmap-nqos-c)# mtu 9216
N7706A(config-pmap-nqos-c)# class type network-qos c-nq-7e-4q8q-ndropfcoe
N7706A(config-pmap-nqos-c)# mtu 2112
N7706A(config-pmap-nqos-c)# system qos
N7706-A(config-sys-qos)# service-policy type CUSTOM-nq-7e-4q8q-policy
!
N7706-A# sh policy-map system type network-qos
policy-map type network-qos CUSTOM-nq-7e-4q8q-policy template 7e-4q8q
class type network-qos c-nq-7e-4q8q-drop
congestion-control tail-drop
mtu 9216
class type network-qos c-nq-7e-4q8q-ndrop-fcoe
pause
mtu 2112
BRKDCN-1100
56
N7K network-qos Templates
CoS 6 & 7 reserved for

control traffic (nonconfigurable)
CoS 3 reserved for
FCoE
Template
Drop CoS
Priority
NoDrop CoS
default-nq8e-policy
0,1,2,3,4,5,6,7
5,6,7
default-nq7e-policy
0,1,2,4,5,6,7
5,6,7
default-nq6e-policy
0,1,2,5,6,7
5,6,7
3,4
default-nq4e-policy
0,5,6,7
5,6,7
1,2,3,4
BRKDCN-1100
57
Nexus 7K
Layer 3 Interfaces
Configured directly on any L3

interface: L3 ports, subinterfaces,
SVIs, port-channels, port-channel
subinterfaces
Parent port-channel interface must have

an equal or greater size
Values can differentiate among L3

interfaces with up to 64 settings per
module
N7706-A(config)# int po10

N7706-A(config-if)# int po10.100
N7706-A(config-if)# sh int po10
port-channel10 is up
admin state is up
Hardware: Port-Channel, address:
Description: 7K-A Agg
Encapsulation ARPA, medium is broadcast
Port mode is routed
N7706-A(config-if)# sh int po10.100
port-channel10.100 is up
admin state is up, [parent interface is port-channel10]
Hardware: Port-Channel, address:
Internet Address is 10.1.1.1/30
Encapsulation 802.1Q Virtual LAN, Vlan ID 100, medium is broadcast
Port mode is routed
BRKDCN-1100
58
Nexus 7K Verification & Troubleshooting
Software verification of port

interface MTU
N7706-A# sh system internal ethpm info interface po10 | i

MTU
medium (broadcast), snmp trap(on), MTU(9000),
N7706-A# sh system internal qos network-qos hw-config | i MTU|slot|CoS

slot 1
UF Configs for CoS 0
MTU
= 1500 [FCoE: No]
MTU
= 1500 [FCoE: No]
MTU
= 1500 [FCoE: No]
MTU
= 2112 [FCoE: Yes]
MTU
= 1500 [FCoE: No]
Inidicates whether port is
MTU
= 1500 [FCoE: No]
using intf level MTU or
inheriting the MTU from the
MTU
= 1500 [FCoE: No]
Virtual Link
MTU
= 1500 [FCoE: No]
Interface
Config Oper(VLs) MTU (value)
Ethernet1/1 Auto Off(0) Port-MTU (9000)
Ethernet1/4 Auto Off(0) VL-MTU
(-)
(-)
(-)
BRKDCN-1100
59
Nexus 7K Verification & Troubleshooting

A configurable rate-limit for packets needing to be
fragmented may be configured; by default it is 500 pps
N7706-A# sh hardware rate-limiter layer-3 mtu
Units for Config: packets per second
Allowed, Dropped & Total: aggregated since last clear counters
rl-1: STP and Fabricpath-ISIS
rl-2: L3-ISIS and OTV-ISIS
rl-3: UDLD, LACP, CDP and LLDP
rl-4: Q-in-Q and ARP request
rl-5: IGMP, NTP, DHCP-Snoop, Port-Security, Mgmt and Copy traffic
Equals 9038 in decimal

(internal header
accommodation)
N7706-A-Admin# attach module 1
Attaching to module 1 ...
To exit type exit, to abort type $.
module-1# sh hardware internal mac port 1 qos | i MTU
Port MTU (No dropping action): 9038
module-1# sh hardware internal mac port 1 sw-state | b MTU

# Port MTU #
port adj_mtu=0x26 last_mtu=0x2316 qos_mtu=0x0 pm_mtu=0x234e
vl0 adj_mtu=0x16 last_mtu=0x2328 qos_mtu=0x2328 pm_mtu=0x2600
Module: 1
Rate-limiter PG Multiplier: 1.00
R-L Class
Config
Allowed
Dropped
Total
+------------------+--------+---------------+---------------+-----------------+
L3 mtu
500
106
0
106
Any packets exceeding the ratelimit will be dropped and counted
BRKDCN-1100
60
Nexus 9000 (Standalone)

MTU Overview
Similar to 7K
Configure MTU for L3 interfaces

directly on the interface
Interface-level MTU for L2 interfaces is
supported (1500 or system
jumbomtu <size>)
System policies (network-qos) may

also be used to apply to all L2
interfaces
BRKDCN-1100
61
Nexus 5500 & 5600

Architecture Overview
Switching paradigms:
Cut-through switch [default for 10G]: frames exceeding the

MTU are truncated and result in eventual CRC errors
Store-and-forward: frames are dropped when MTU is
exceeded
Layer 3 capabilities are integrated at line rate for 5600

series and are available with L3 module for 5500 series
BRKDCN-1100
62
Nexus 5500 & 5600

Switching Modes
When 10G
Ethernet is used
ingress, cutthrough switching
is used by default
Src Int
Dst Int
Switching Mode
10G Ethernet
10G Ethernet
Cut-through
10G Ethernet
1G Ethernet
Cut-through
1G Ethernet
1G Ethernet
Store-and-Forward
1G Ethernet
10G Ethernet
Store-and-Forward
FCoE
Fibre Channel
Cut-through
Fibre Channel
FCoE
Store-and-Forward
Fibre Channel
Fibre Channel
Store-and-Forward
FCoE
FCoE
Cut-through
BRKDCN-1100
63
Nexus 5500 & 5600

Layer 2 Interfaces
Must use a nework-qos policy-map
Per interface MTU not supported

One system-wide policy
qos-groups uniquely identify the system

classes internally
MTU may be configured

on a per class basis
5 configurable, 1 default class and 2 reserved for

control traffic
System defaults
system jumbomtu 9216; configurable from
2158 to 9216 bytes
Default network-qos policy cannot be modified
BRKDCN-1100
64
Nexus 5500 & 5600

Layer 2 MTU Configuration
N5672-A(config)# policy-map type network-qos jumbo-nq-policy
N5672-A(config-pmap-nq)# class type network-qos class-fcoe
Both commands
N5672-A(config-pmap-nq-c)# pause no-drop
required for
N5672-A(config-pmap-nq-c)# mtu 2158
FCoE class
N5672-A(config-pmap-nq-c)# class type network-qos class-default
N5672-A(config-pmap-nq-c)# mtu 9000
N5672-A(config)# system qos
N5672-A(config-sys-qos)# service-policy type network-qos jumbo-nq-policy
N5672-A# sh policy-map system
===============================
policy-map type network-qos jumbo-nq-policy
class type network-qos class-fcoe
match qos-group 1
pause no-drop
mtu 2158
class type network-qos class-default
match qos-group 0
mtu 9000
BRKDCN-1100
65
Nexus 5500 & 5600

L2 MTU verification
N5672-A# sh int e1/37
Ethernet1/37 is up
Dedicated Interface
Description: Dell VRTX Eth8/1
Encapsulation ARPA
Port mode is trunk
full-duplex, 10 Gb/s, media type is 10G
<output omitted>
30 seconds input rate 93872 bits/sec, 38 packets/sec
30 seconds output rate 170712 bits/sec, 101 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 80.43 Kbps, 15 pps; output rate 154.54
Kbps, 78 pps
RX
38255265 unicast packets 256631 multicast
packets 418643 broadcast packets
38930539 input packets 23961814506 bytes
11702772 jumbo packets 0 storm suppression bytes
<output omitted>
TX
71224821 unicast packets 89774097 multicast
packets 4865803 broadcast packets
165864721 output packets 84561892595 bytes
45431246 jumbo packets
<output omitted>
N5672-A# sh queuing int e1/37

Ethernet1/37 queuing information:
TX Queuing
qos-group sched-type oper-bandwidth
0
WRR
50
1
WRR
50
RX Queuing
qos-group 0
q-size: 100160, HW MTU: 9000 (9000 configured)
drop-type: drop, xon: 0, xoff: 0
Statistics:
Pkts received over the port
: 449082142
Ucast pkts sent to the cross-bar
: 448848703
Mcast pkts sent to the cross-bar
: 233439
Ucast pkts received from the cross-bar : 244633695
Pkts sent to the port
: 317668644
Pkts discarded on ingress
: 0
Per-priority-pause status
: Rx (Inactive), Tx (Inactive)
qos-group 1
q-size: 165120, HW MTU: 2158 (2158 configured)
drop-type: no-drop, xon: 62720, xoff: 88320
Statistics:
Pkts received over the port
: 3446851283
Ucast pkts sent to the cross-bar
: 3446902243
Mcast pkts sent to the cross-bar
: 0
Ucast pkts received from the cross-bar : 2610115540
Pkts sent to the port
: 2610115564
Pkts discarded on ingress
: 0
Per-priority-pause status
: Rx (Active), Tx (Active)
BRKDCN-1100
66
Nexus 5500 & 5600 Troubleshooting
Using cut-through forwarding, frame is passed to egress ASIC once DA is

parsed
5K is receiving 9216 byte

frames while configured for
only 1500 MTU
Excellent command for

seeing packet sizes / types
on an interface
N56128-A# sh hardware internal bigsur port e1/48 counters rx
Port 10gb1/48 MAC statistics:

MAC Statistic
| Value
| Diff
| Rate
|
-------------------------------+-----------------------------------+----------------------+--------------+
<snip>
RX_PKT_SIZE_IS_8192_TO_9216
|181538
|181538
|0
N56128-A# sh hardware internal bigsur all-ports
Determine which bigsur ASIC

the interface belongs to...
Switch is detecting
MTU violations;
packets are truncated
Bigsur Port Info:

Port
|asic|inst|inst|
name
|idx |slot|asic|eport|logi|flag|adm|opr|if_index|diag|ucVer
---------+----+----+----+-----+----+----+---+---+--------+----+----<snip>
xgb1/48 |3
|0
|3
|11 p |47 |b3 |en |up |1a02f000|pass| 0.00
N56128-A# sh hardware internal bigsur asic 3 counters interrupt | i mtu
Slot 0 Bigsur3 interrupt statistics:
Interrupt name
|Count
|ThresRch|ThresCnt|Ivls
-----------------------------------------------+--------+--------+--------+---<snip>
big_bmin_cl1_INT_p1_err_ig_mtu_vio
|30
|0
|30
|0
BRKDCN-1100
67
Nexus 5500 & 5600 Troubleshooting
On the egress 5K interface;

output errors are also
shown in show interface
N56128-A# sh hardware internal bigsur port e1/4 counters rx

Port 10gb1/4 MAC statistics:
MAC Statistic
| Value
| Diff
| Rate
|
-------------------------------+-----------------------------------+----------------------+--------------+
<snip>
TX_PKT_FRAME ERROR
|181538
|181538
|0
N56128-A# sh hardware internal bigsur asic 0 counters interrupt | i crc
Frames are CRC stomped
Slot 0 Bigsur0 interrupt statistics:

Interrupt name
|Count
|ThresRch|ThresCnt|Ivls
-----------------------------------------------+--------+--------+--------+---<snip>
big_fwe_psr1_P2_INT_pkt_err_eth_crc_stomp
|30
|0
|30
|0
BRKDCN-1100
68
Nexus 5500 & 5600

L3 Interfaces
Set MTU directly on the interface
Use mtu <size> syntax under the interface (physical, SVI,

Port-Channel)
Can only support an MTU of up to 9192 on L3 interfaces
Caused by 24 byte switch internal header

Implies that other devices should be adjusted to
accommodate if L3 is needed on 5K
BRKDCN-1100
69
Nexus 3000, 3100, 3500 Series
Nexus 3000
Nexus 3100, 3500
Per-port MTU configuration

supported for L2 and L3 interfaces
May use to network-qos policy to

affect all ports
network-qos policy must be used
BRKDCN-1100
70
Nexus 2200 & 2300FEX

Architecture Overview
Jumbo frame support is dependent

on the parent switch configuration
Auto-inherit of network-qos policy
No local switching on FEX
FEX links use a VNTag (6 bytes) & an

802.1Q tag (4 bytes)
Programed when NIFs are configured
BRKDCN-1100
71
Nexus 2200 & 2300FEX

7K Parent
6.2(2)+:All FEX MTU controlled by

network-qos policy
Prior releases only MTU modification was

needed on FPC
Create a new network-qos policy in the

Default or Admin VDC
Changes are automatically pushed

down to the FEX
Example enables jumbo

frames for all traffic classes
N7706-Admin-A(config)# class-map type network-qos
match-any class-nq-8e
N7706-Admin-A(config-cmap-nqos)# match cos 0-7
N7706-Admin-A(config)# policy-map type network-qos
jumbo-nq-8e template 8e-4q8q
N7706-Admin-A(config-pmap-nqos)# class-map type
network-qos class-nq-8e
N7706-Admin-A(config-pmap-nqos-c)# mtu 9000
N7706-Admin-A(config-pmap-nqos-c)# congestion taildrop
N7706-Admin-A(config)# system qos
N7706-Admin-A(config-sys-qos)# service-policy type
network-qos jumbo-nq-8e
BRKDCN-1100
72
Nexus 1000v
VMware (N1K 5.x)
system jumbomtu 9000 is a fixed global setting
For jumbo frame forwarding, MTU config needed on eth port-profiles OR

interfaces
N1kv-VSM-A(config)# port-profile type eth Uplink_PP
N1kv-VSM-A(config-port-prof)# mtu 9000
Nexus1000v-VSM-A(config)# int e3/1

Nexus1000v-VSM-A(config-if)# mtu 9000
Confirm on ESXi host:
[root@UCSC240:~] esxcfg-nics -l
Name
PCI
Driver
vmnic0 0000:02:00.0 igb
vmnic1 0000:02:00.1 igb
Link Speed
Up
1000Mbps
Up
1000Mbps
Duplex MAC Address

MTU
Full
a4:93:4c:aa:fb:6e 9000
Full
a4:93:4c:aa:fb:6f 9000
Description
Intel Corporation I350 Gigabit Network Connection
Intel Corporation I350 Gigabit Network Connection
BRKDCN-1100
73
Nexus 1000v
Microsoft Hyper-V
No management over MTU settings

within the virtual modules
Physical adapters added to 1000v

should have desired MTU
Modify via the MS adapter properties,

UCSM, or ILO where applicable
Value should not be changed after NIC is
added to the switch
MTU value not reflected on VSM/VEM

interfaces; verify on host with
.\vemcmd.exe show attach info
---------------------------------------------LTL:
17
---------------------------------------------Port ID:
4
NIC Index:
2
NIC Instance ID: {7C3A395E-ADDF-41B8-BE39D05BC73C4699}
MAC address:
a0:36:9f:6a:c8:92
Port profile:
UPLINK-PP
Port profile GUID: 6AF1BB76-F87A-4EAB-94808420AFB6CD02
Network GUID:
VM/NIC name:
Intel(R) Ethernet 10G 2P X520
Adapter
VM UUID:
NIC UUID:
MTU:
9000
Link state:
UP
Duplex:
Full
Tx speed:
1410065408
Rx speed:
1410065408
Autoneg:
Enabled
Link Params pending: No
Speed Capability 0x10
Duplex Capability 0x4
BRKDCN-1100
74
UCS Manager
Fabric Interconnect Processing
Host
Ingress
Policing
Violation =
truncate or drop
Threshold reached =
-Tail drop for drop classes
-Pause for lossless
MTU Check
Per-class
buffer monitor
8 multicast
queues
Egress Traffic
Traffic
Classification
Ingress Traffic
Trusted CoS
L2/L3/L4 Details
Host
Marking
CoS
Egress
Scheduler
Strict Priority or
DWRR Scheduler
Egress
Queues
Unicast VOQ
(8 per egress interface)
BRKDCN-1100
75
UCS Manager
QoS System Classes
Match criteria
Manually enter desired MTU

for each class
normal: 1500 bytes
Auto-created / hard-set with

vHBA creation;
fc: 2158 bytes
BRKDCN-1100
76
UCS Manager
Assigning to vNICs
Specify the MTU on the vNIC;

informs the host network adapter of
MTU value to be used
vNICs can be assigned
specific priority; otherwise
will default to Best Effort
9000
6248-FI-A# scope service-profile server 1/1

6248-FI-A /org/service-profile# show vnic
vNIC:
Name
Fabric ID Dynamic MAC Addr
Virtualization Preference
-------------- ---------- ----------------- ------------------------1-host-vNIC-A
A
00:25:B5:0A:00:01 NONE
1-host-vNIC-B
B
00:25:B5:0B:00:01 NONE
6248-FI-A /org/service-profile# show vnic 1-host-eth-1 detail
vNIC:
<output omitted>
Host Interface Ethernet MTU: 9000
BRKDCN-1100
77
End-to-End Testing
Verification across platforms

Linux: ping 192.168.1.5 M do s 8972 OR tracepath
ESXi: vmkping d s 8972 192.168.1.5
Windows: mturoute OR
Subtract the IP header &

ICMP header (28 bytes)
from the ping size
Be aware of default CoPP policy that rate-limits jumbo ICMP packets

destined to the local Nexus switch
BRKDCN-1100
78
Summary
In Conclusion
Not all traffic is created equal
Know the flow
Align the stars
BRKDCN-1100
80
Complete Your Online Session Evaluation
Give us your feedback to be

entered into a Daily Survey
Drawing. A daily winner will
receive a $750 Amazon gift card.
Complete your session surveys

through the Cisco Live mobile
app or from the Session Catalog
on CiscoLive.com/us.
Dont forget: Cisco Live sessions will be available
for viewing on-demand after the event at
CiscoLive.com/Online
BRKDCN-1100
81
Continue Your Education
Demos in the Cisco campus
Walk-in Self-Paced Labs
Lunch & Learn
Meet the Engineer 1:1 meetings
Related sessions
Presentation ID
82
Thank you

BRKDCN-1100 - MTU Considerations and Implications in The Data Center (2016 Las Vegas)

Transféré par

Informations du document

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

BRKDCN-1100 - MTU Considerations and Implications in The Data Center (2016 Las Vegas)

Transféré par

Droits d'auteur :

Formats disponibles

Size Matters: MTU Considerations

and Implications in the Data Center

Overview of MTU within the Data Center

Sizing across multiple protocols

MTU interaction with Data Center technologies

Ins & Outs of MTU related configuration

Overview of MTU within the

The Data Center Evolution

STP Tiered Design

Full Link Utilization

Host & Network Overlays

Separate SAN Infrastructure

What is the maximum size of a

Any payload size greater than

Does not include Ethernet headers

Nexus & UCS support payloads of

Why the Draw to Jumbo Frames?

A 9000 byte frame results in up to 4 context switches and 4 copies to message

Improved forwarding and buffer utilization from network devices

Finding the Right Size

1. Host sends 9000 byte packet to discover

Transmit with new MTU value

Effects of MTU Mismatches

Issues with dynamic routing

The Issue(s) with Fragmentation...

Breaks down large IP datagrams into smaller datagrams so

Fragmentation introduces more router load

Only initial fragment contains upper-level protocol headers

Non-initial fragments will match only source/destination IP

The Bigger The Better?

Jumbo frame support is not widespread

North / South and inter-DC traffic should be

Doesnt play well with real-time traffic

More prone to performance issues in terms

Its Good Where It Makes Sense

East / West traffic; the busy parts

Private WAN where there is control

All stars need to align

Sizing Across Multiple Protocols

Making It All Flow...

NX-OS System QoS Policies

QoS Policy Types

CoS, DSCP, ACL, IPP,

Marking, policing, mutation

MTU, No-drop (lossless Ethernet)

Priority, queueing, buffering,

Type Network-QoS Policies

One per system that is applied to the

Controls the shape of the traffic and

Exception is native FC interfaces

Ability for varying MTU values among

Why use CoS?

MTU may be determined based on:

Switched traffic exceeding the MTU is truncated or dropped

No fragmentation supported on routers!

Must rely on PMTUD, consistent MTU scheme, RA messages

Hosts are responsible for fragmentation

NX-OS has one interface-level MTU command: mtu <size>

Influences IPv4 & IPv6 alike

Instructs the host to use an MTU of

N7706-A(config-if)# ipv6 nd mtu 1480

Block-level storage access over IP

Can co-exist with LAN infrastructure