Path MTU Discovery: inside out

Eric Vyncke
Field Distinguished Engineer
Cisco Systems evyncke@cisco.com +32 2 778.4677
Evyncke Path MTU Discovery

2002, Cisco Systems, Inc. All rights reserved.

Introduction

The slides mainly apply to IOS and possibly to the PIX except when specified.

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

Agenda

IP Fragmentation
Path MTU Discovery Impact of GRE Impact of IPSec Impact of GRE followed by IPSec

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

IP: Packet Format

0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |Version| IHL |Type of Service| Total Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Identification |Flags| Fragment Offset | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Time to Live | Protocol | Header Checksum | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Source Address | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Destination Address | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Options | Padding | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Internet Datagram Header

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

IP Fragmentation Fields

All fragments of the original datagram keep the original identification fragments are identified by fragment offset Flags can contain:
MF: More Fragments (set on all but last fragments) DF: Do not Fragment (prevent fragmentation)
Evyncke PMTUD
2002, Cisco Systems, Inc. All rights reserved.

IP Normal Fragmentation (Cont.)

Before fragmentation:
ID=x,TL=1300,FO=0,MF=0

data length 1280 IP data

IP Header

After fragmentation (MTU = 500):

ID=xTL=500,FO=0,MF=1

data length 480 data length 480

ID=x,TL=500,FO=480,MF=1

ID=x;TL=340,FO=960,MF=0

data length 320

9

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

Impact of Fragmentation
Fragmentation should be avoided at all costs:
Increased packet loss rate Waste of CPU and memory on nodes doing reassembly

Some low cost PAT box (linksys, Netgear) are unable to process fragments communication is simply broken this is becoming an issue for IKE (udp/500) Routers are inefficient at IP packet reassembly
Must allocate largest memory buffer Always process-switched
Evyncke PMTUD
2002, Cisco Systems, Inc. All rights reserved.

11

Agenda

IP Fragmentation
Path MTU Discovery Impact of GRE Impact of IPSec Impact of GRE followed by IPSec

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

12

Path MTU Discovery (PMTUD)

Path MTU Discovery tries to measure the smallest MTU on the path and the originator will then send datagrams smaller PMTUD is only used by TCP and some tunnel protocols (Ex: GRE, IPSec, IPinIP).

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

13

Normal Path MTU Discovery/1

MTU=1500 S R1 MTU=1500 R2 MTU=1000 R3 MTU=1500 R4 MTU=1500 C

Described in RFC 1191

The smallest MTU is 1000 bytes S should not send IP datagrams larger than 1000 bytes To discover the smallest Path MTU between S and C, S will send an IP datagram (size its MTU = 1500) with DF bit set

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

14

Normal Path MTU Discovery/2

MTU=1500 S R1 MTU=1500 R2 MTU=1000 R3 MTU=1500 R4 MTU=1500 C

1.

2.

3.

R2 cannot forward the datagram because MTU=1000 would mean fragmentation; but fragmentation is prevented by the DF bit. R2 sends an ICMP unreachable 3/4 to the source S. ICMP contains the MTU and part of the failed IP datagram

1. 2.

Upon receive of the ICMP unreachable, S update an internal table S will only send datagram <= 1000 to C

This works fine with notably Windows NT/2000 servers

Evyncke PMTUD
2002, Cisco Systems, Inc. All rights reserved.

15

Issues with PMTUD

PMTUD uses a negative signalling (ICMP in case of trouble):
No news is good news

But, ICMP are:

Blocked by dumb firewalls or dumb security policies (should allow code 3 type 4 inbound) Ignored by load balancers (including Cisco ones but this is being fixed)

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

16

Hacking Windows.

PMTUD can be disabled on Windows servers through registry

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\ Services\Tcpip\Parameters\EnablePMTUDiscover y = 0

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

17

Adjust TCP MSS to solve broken PMTUD

Blocked ICMP are reality TCP MSS (Maximum Send Segment) = maximum TCP payload. MSS can be configured to MSS+40 <= smallest MTU to avoid fragmentation. Obviously works only for TCP

IOS (with CSCds69577 in 12.3T)

ip tcp adjust-mss 1400

PIX
sysopt tcpmss 1400

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

18

Effect of reducing MSS

Sysopt tcpmss 960 MTU=1500 S R1 MTU=1500 R2 MTU=1000 R3 PIX MTU=1500 C

1. 2. 3.

S will never send IP datagram > 1000 bytes No fragmentation will occur If S is PMTUD, as no fragmentation is done, no ICMP will be generated

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

19

Agenda

IP Fragmentation
Path MTU Discovery Impact of GRE Impact of IPSec Impact of GRE followed by IPSec

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

20

Normal GRE Encapsulation

Original IP datagram (before forwarding) Original IP header
20 bytes

IP payload

GRE encapsulation (after forwarding to a GRE tunnel) GRE header

Protocol=800
4 bytes

Original IP header
20 bytes

IP payload

GRE packet with new IP header: protocol 47 (forwarded using new IP dst)
External IP header GRE header
DF=0, protocol=47
20 bytes

Protocol=800
4 bytes

Original IP header
20 bytes

IP payload

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

22

GRE and fragmentation

The MTU of the GRE tunnel interface is by default the MTU of the physical interface 24 bytes
E.g.: for a Ethernet GRE, MTU is 1476 bytes

Per default DF bit of GRE packet is set to 0

The MTU of the GRE tunnel interface can be set with

ip mtu
Original packet DF=0 Is forwarded through GRE

YES

Original packet <= GRE MTU ?

NO

Encapsulate the original Packet in a single GRE packet

Evyncke PMTUD
2002, Cisco Systems, Inc. All rights reserved.

1. 2.

Fragment the original packet Encapsulate all fragments in different GRE packets
23

GRE Encapsulation with Fragmentation /1

Original IP datagram (total length = 1500) Default behaviour: MTU of GRE tunnel is set to 1476 => Original IP is fragmented Original IP header
FO=0,MF=0,DF=0,ID=x
20 bytes

IP payload
1480 bytes

2 GRE packets (total length <= physical MTU = 1500) External IP header
FO=0, MF=0, ID=1
20 bytes

GRE header
4 bytes

Original IP header
FO=0,MF=1,DF=0,ID=x
20 bytes

IP payload
1456 bytes

External IP header
FO=0, MF=0, ID=2
20 bytes

GRE header
4 bytes

Original IP header
FO=1456, MF=0, ID=x
20 bytes

IP payload
24 bytes

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

24

GRE Encapsulation with Fragmentation /2

Non default behaviour: MTU of GRE tunnel is configured to 1500 Interface tunnel ip mtu 1500 => Original IP is not fragmented Original IP datagram (total length = 1500) Original IP header
FO=0, MF=0, ID=x
20 bytes

IP payload
1480 bytes

After encapsulation GRE header Original IP header

FO=0, MF=0, ID=x
20 bytes 4 bytes

IP payload
1480 bytes

External IP header
FO=0, MF=1, ID=1
20 bytes

GRE header
4 bytes

Original IP header
FO=0, MF=0, ID=x
20 bytes

IP payload
1456 bytes

1 GRE packet in two fragments

Evyncke PMTUD
2002, Cisco Systems, Inc. All rights reserved.

External IP header
FO=1480, MF=0, ID=1
20 bytes

IP payload
24 bytes
25

Normal IP fragmentation & GRE

MTU=1500-24=1476
MTU=1500 S R1 MTU=1500 R2 MTU=1000 R3 MTU=1500 R4 MTU=1500 C

Host C reassembles the 2 fragments 1. 2. R1 fragments Sends 2 GRE packets with DF=0 1. R3 reassembles to reconstruct 1st GRE packet Both GRE packets are decapsulated The 2 fragments of original IP datagram are forwarded
26

1.
2.

1st GRE packet is too large and is further fragmented 2nd GRE packet is forwarded unchanged
2002, Cisco Systems, Inc. All rights reserved.

2. 3.

Evyncke PMTUD

Normal IP fragmentation and GRE: conclusion

Nothing is broken: data traffic is flowing

But, 2 fragmentations occur

At tunnel entry point On the tunnel path

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

27

PMTUD & GRE

MTU=1500-24=1476
MTU=1500 S R1 MTU=1500 R2 MTU=1000 R3 MTU=1500 R4 MTU=1500 C

1. 2.

R1 needs to fragment but original IP has DF=1 R1 sends ICMP unreachable to S GRE packet is too large and is further fragmented (DF=0)

1.

2.

Upon receive of ICMP unreachable, S will send maximum 1476 bytes 2nd IP packet is 1476 bytes long

1. 2. 3.

R4 reassembles to reconstruct the GRE packet (R4 is the destination of GRE packets) GRE packet is decapsulated The original IP datagram is forwarded
28

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

PMTUD and GRE: conclusion

Nothing is broken: data traffic is flowing

But, 1 fragmentation still occurs
On the tunnel path

Reason: the GRE packets has DF=0 per default

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

29

The GRE MTU 1500: trick

Configuring manually the GRE MTU to 1500

No fragmentation before GRE encapsulation (no ICMP sent) The GRE packet has DF=0 and can be fragmented

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

30

PMTUD with GRE MTU 1500

Non default behaviour: MTU of GRE tunnel is configured to 1500 Interface tunnel ip mtu 1500 => Original IP is not fragmented Original IP datagram (total length = 1500) Original IP header
FO=0,MF=0,DF=1,ID=x
20 bytes

IP payload
1480 bytes

After encapsulation GRE header Original IP header

4 bytes 20 bytes

FO=0,MF=0,DF=1,ID=x

IP payload
1480 bytes

External IP header
FO=0,MF=1,DF=0,ID=1
20 bytes

GRE header
4 bytes

Original IP header
FO=0,MF=0,DF=1,ID=x
20 bytes

IP payload
1456 bytes

1 GRE packet in two fragments

Evyncke PMTUD
2002, Cisco Systems, Inc. All rights reserved.

External IP header
FO=1480,MF=0,DF=0,ID=1
20 bytes

IP payload
24 bytes
31

PMTUD with GRE MTU 1500

MTU=1500
MTU=1500 S R1 MTU=1500 R2 MTU=1000 R3 MTU=1500 R4 MTU=1500 C

1. 2.

R1 uses 1 GRE packet with DF=0 R1 fragments the GRE packet 1. R3 reassembles to reconstruct the GRE packet The original IP datagram is forwarded

1.
2.

1st GRE fragment is too large and is further fragmented 2nd GRE fragment is forwarded unchanged
2002, Cisco Systems, Inc. All rights reserved.

2.

Evyncke PMTUD

32

PMTUD with GRE MTU 1500: Conclusion

Configuring manually the GRE MTU to 1500

No fragmentation before GRE encapsulation (no ICMP sent)
The GRE packet is fragmented Works even if ICMP are filtered

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

33

Tunnel PMTUD for GRE tunnel

Since 12.0(5)T (but still undocumented)
tunnel path-mtu-discovery

Changes
1. The DF bit is copied from original IP packet 2. GRE tunnel endpoints listen to ICMP unreachable 3. When GRE tunnel endpoint gets an ICMP unreachable, the MTU of GRE tunnel is dynamically updated

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

34

tunnel path-mtu-discovery /1
MTU=1500-24=1476
MTU=1500 S R1 MTU=1500 R2 MTU=1000 R3 MTU=1500 R4 MTU=1500 C

1. 2.

R1 needs to fragment but original IP has DF=1 R1 sends ICMP unreachable to S 1. 2. Upon receive of ICMP unreachable, S will send maximum 1476 bytes 2nd IP packet is 1476 bytes long 1. 2. 3. GRE packet is too large and cannot be fragmented (DF=1) Packet is dropped ICMP sent to R1 (source of GRE packet)

Upon receive of ICMP unreachable, R1 will update GRE MTU to 1000-24=976 bytes
Evyncke PMTUD
2002, Cisco Systems, Inc. All rights reserved.

35

tunnel path-mtu-discovery /2
show interface tunnel 0 The Physical MTU (badly Tunnel0 is up, line protocol is up displayed) is actually 1476 Hardware is Tunnel Description: Virtual interface to router C Internet address is 192.168.100.2/30 MTU 1514 bytes, BW 9 Kbit, DLY 500000 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation TUNNEL, loopback not set Keepalive not set Tunnel source 194.194.194.1, destination 193.193.193.1 Tunnel protocol/transport GRE/IP, key disabled, Checksumming of packets disabled, fast tunneling enabled Path MTU Discovery, ager 10 mins, MTU 976, expires 00:09:36

The path MTU as updated by ICMP

Evyncke PMTUD
2002, Cisco Systems, Inc. All rights reserved.

36

tunnel path-mtu-discovery /3
MTU is now set to 976
MTU=1500 S R1 MTU=1500 R2 MTU=1000 R3 MTU=1500 R4 MTU=1500 C

S retries with packet length 1476 bytes & DF=1 1. 2. R1 needs to fragment but original IP has DF=1 R1 sends ICMP unreachable to S

1.
2.

Upon receive of ICMP unreachable, S will send maximum 976 bytes 2nd IP packet is 976 bytes long

No more fragmentation
Evyncke PMTUD
2002, Cisco Systems, Inc. All rights reserved.

37

tunnel path-mtu-discovery: conclusion

With tunnel path-mtu-discovery
GRE MTU is dynamically learned No more fragmentation occur One IP packet is lost Relies on ICMP messages received by S and R1 (beware of dumb security policies and load balancers)

Without it, use ip mtu to set GRE MTU to the lowest MTU in the path - 24

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

38

Tactics for GRE & Fragmentation

If ICMP are filtered, only solution is setting MTU to 1500
Normal IP packet
Entrance of GRE tunnel Default MTU Fragment original: 2 GRE packets Fragment GRE packet Fragment original: 2 GRE packets In the GRE tunnel path Fragment GRE packet Fragment the GRE fragment Fragment GRE packet

IP packet with DF=1

Entrance of GRE tunnel Adjust senders MTU Fragment GRE packet Adjust senders MTU In the GRE tunnel path Fragment GRE packet Fragment the GRE fragment Adjust GRE tunnel MTU

Ip mtu 1500

Tunnel-pathmtu-discovery

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

39

IPinIP: RFC 2003

IPinIP is quite similar to GRE IPinIP is used by IPSec tunnel mode Seems to get some momentum in IETF (PPVPN WG)

All what is described for GRE also works for IPinIP (at least for IOS)

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

40

IPinIP Encapsulation
IPinIP is RFC2003 Standards Track by IBM Uses protocol 4 Only works for IP Used by IPsec tunnel mode Almost identical to GRE in IOS Original IP datagram (before forwarding) Original IP header
20 bytes

IP payload

IPinIP encapsulation (after forwarding to a IPinIP tunnel) Original IP header

20 bytes

IP payload

IPinIP packet with new IP header: protocol 4 (forwarded using new IP dst)
External IP header
DF=0, protocol=4
20 bytes

Original IP header
20 bytes

IP payload

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

41

Agenda

IP Fragmentation
Path MTU Discovery Impact of GRE Impact of IPSec Impact of GRE followed by IPSec

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

42

Normal Tunnel Mode IPSec Encapsulation

Default behaviour: DF bit is copied to the external IP header
Original IP datagram (after forwarding)

Original IP header
DF=x
20 bytes

IP payload

IPSec ESP without ESP auth encapsulation (after encapsulation) ESP header
16 bytes

Original IP header
20 bytes

IP payload

ESP trailer
2-10 bytes

IPSec packet with new IP header: protocol 50/51 (on the wire)
External IP header
DF=x
20 bytes

ESP header
16 bytes

Original IP header
20 bytes

IP payload

ESP trailer
2-10 bytes

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

44

Tunnel Mode IPSec and Fragmentation

Original IP datagram (after forwarding) Original IP header
FO=0, ID=x, DF=0
20 bytes

IP payload
1480 bytes

IPSec ESP tunnel mode packet with new IP header (after encapsulation) ESP Original IP header FO=0,MF=0,ID=y,DF=0 header FO=0, ID=x, DF=0 External IP header
20 bytes 16 bytes 20 bytes

IP payload
1480 bytes

ESP trailer
8 bytes

ESP Original IP header FO=0,MF=1,ID=y,DF=0 header FO=0, ID=x, DF=0 External IP header
20 bytes 16 bytes 20 bytes

IP payload
1444 bytes

1 IPSec packet in two fragments

(on the wire)

External IP header
FO=1480,MF=0,ID=y,DF=0
20 bytes

IP payload
36 bytes

ESP trailer
8 bytes
45

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

Difference between GRE and IPSec Tunnel Mode

GRE fragments before encapsulation

1 large IP packet will result in 2 GRE packets Original IP packet is fragmented

IPSec Tunnel Mode fragments after encapsulation

1 large IP packet will result in 1 IPSec packet (in two fragments) Encrypted packet is fragmented
Evyncke PMTUD
2002, Cisco Systems, Inc. All rights reserved.

47

Normal IP fragmentation & IPSec

MTU=1500 S R1 MTU=1500 R2 MTU=1000 R3 MTU=1500 R4 MTU=1500 C

1. 2.

Encapsulate into 1 IPSec packet Sends 1 IPSec packet in 2 fragments 1. R3 reasembles to reconstruct the IPSec packet IPSec packet is decapsulated The original IP datagram is forwarded
49

1.
2.

1st fragment is too large and is further fragmented 2nd fragment is forwarded unchanged
2002, Cisco Systems, Inc. All rights reserved.

2. 3.

Evyncke PMTUD

Normal IP fragmentation and IPSec: conclusion

Nothing is broken: data traffic is flowing

But, 2 fragmentations occur

At IPSec tunnel entry point On the IPSec tunnel path

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

50

IPSec and PMTUD

IPSec encapsulation copies the DF bit to the external header (compliance with RFC 2401) IPSec keeps track of the path MTU of the tunnel
Initialised to physical MTU

Actual MTU before fragmenting is physical MTU 46

(20 for external IP, 16 for ESP header, max 10 for ESP trailer)

Dynamically updated with ICMP unreachable

NB: this was broken in a lot of IOS images

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

51

Doing PMTUD for IPSec/1

Path MTU = Media MTU = 1500
MTU=1500 S R1 MTU=1500 R2 MTU=1000 R3 MTU=1500 R4 MTU=1500 C

1. 2.

R1 needs to fragment but original IP has DF=1 R1 sends ICMP unreachable to S (using Path MTU 46) 1. 2. Upon receive of ICMP unreachable, S will send maximum 1454 bytes 2nd IP packet is 1454 bytes long 1. 2. 3. IPSec packet is too large and cannot be fragmented (DF=1) Packet is dropped ICMP sent to R1 (source of IPSec packet)

Upon receive of ICMP unreachable, R1 will update IPSec Path MTU to 1000 (after encapsulation)
Evyncke PMTUD
2002, Cisco Systems, Inc. All rights reserved.

52

Doing PMTUD for IPSec/2

IPSec Path MTU is now set to 1000
MTU=1500 S R1 MTU=1500 R2 MTU=1000 R3 MTU=1500 R4 MTU=1500 C

S retries with packet length 1454 bytes & DF=1 1. 2. R1 needs to fragment but original IP has DF=1 R1 sends ICMP unreachable to S (using Path MTU 46)

1.
2.

Upon receive of ICMP unreachable, S will send maximum 954 bytes 2nd IP packet is 954 bytes long

No more fragmentation IF AND ONLY IF ICMP messages to R1 and S are not filtered
Evyncke PMTUD
2002, Cisco Systems, Inc. All rights reserved.

53

Normal IPSec & PMTUD: conclusion

With normal IPSec

IPSec tunnel MTU is dynamically learned No more fragmentation occurs One IP packet is lost Relies on ICMP messages received by S and R1 (beware of dumb security policies and load balancers)

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

54

Filtered ICMP Messages

For most VPN over Internet, ICMP messages can be lost
Plain packet drop ICMP dumbly filtered out (security policy) ICMP not sent by Internet router (no icmp unreachable or using private addresses) Most server farms are using load balancers

Workarounds:
1. Set ip mtu on BOTH sides to a low enough MTU 2. Hide PMTUD by always setting DF=0 on external header (requires 12.2(2)T) 3. Use the MSS trick

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

55

Clearing DF: IPSec & PMTUD

Using 12.2(2)T crypto ipsec df-bit clear
Original IP datagram (after forwarding) Original IP header IP payload
FO=0, ID=x, DF=1
20 bytes 1480 bytes

IPSec ESP tunnel mode packet with new IP header (after encapsulation) ESP Original IP header FO=0,MF=0,ID=y,DF=0 header FO=0, ID=x, DF=1 External IP header
20 bytes 16 bytes 20 bytes

IP payload
1480 bytes

ESP trailer
8 bytes

ESP Original IP header FO=0,MF=1,ID=y,DF=0 header FO=0, ID=x, DF=1 External IP header
20 bytes 16 bytes 20 bytes

IP payload
1444 bytes

1 IPSec packet in two fragments

(on the wire)

External IP header
FO=1480,MF=0,ID=y,DF=0
20 bytes

IP payload
36 bytes

ESP trailer
8 bytes
56

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

Clearing DF: IPSec & PMTUD

MTU=1500 S R1 MTU=1500 R2 MTU=1000 R3 MTU=1500 R4 MTU=1500 C

1. 2.

Encapsulate into 1 IPSec packet w/ DF=0 Sends 1 IPSec packet in 2 fragments 1. R3 reassembles to reconstruct the IPSec packet IPSec packet is decapsulated The original IP datagram is forwarded
57

1.
2.

1st fragment is too large and is further fragmented 2nd fragment is forwarded unchanged
2002, Cisco Systems, Inc. All rights reserved.

2. 3.

Evyncke PMTUD

Tactics for IPSec & Fragmentation

If ICMP are filtered, only solution is clearing DF
Normal IP packet
Entrance of IPSec tunnel Clear DF bit Fragment IPSec packet Fragment original: 2 IPSec packets In the IPSec tunnel path Fragment the fragment

IP packet with DF=1

Entrance of IPSec tunnel Fragment IPSec packet In the IPSec tunnel path Fragment the fragment

Ip mtu 1400

Fragment IPSec Adjust senders Fragment IPSec packet MTU packet Adjust IPSec tunnel MTU

default

Fragment IPSec Fragment IPSec Adjust senders packet packet MTU

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

58

VPN 3000 Concentrator

Does not honour the DF bit
Will happily fragment a packet with DF=1 Effect: disable PMTUD outside of the IPSec tunnel

Clear the DF bit when encapsulating

Effect: disable PMTUD along the IPSec tunnel

Possible loss of performance but will always work even if ICMP are filtered
Evyncke PMTUD
2002, Cisco Systems, Inc. All rights reserved.

59

Agenda

IP Fragmentation
Path MTU Discovery Impact of GRE Impact of IPSec Impact of GRE followed by IPSec

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

60

Combination of GRE + IPSec

The powerful combo

GRE: for routing protocols, multicast IPSec in transport mode: confidentiality

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

61

Normal GRE+IPSec Encapsulation

Default behaviour: DF bit is cleared after GRE encapsulation
Original IP datagram (before forwarding)

Original IP header
DF=x
20 bytes

IP payload

GRE encapsulation (after forwarding to a GRE tunnel)

GRE Original IP header

4 bytes 20 bytes

IP payload

GRE+IPSec packet with new IP header (just before transmission) External IP header
DF=0
20 bytes

ESP

GRE Original IP header

20 bytes

IP payload

ESP
2-10 bytes

16 bytes 4 bytes

Note: IPSec ESP without authentication and in transport mode is used.

Evyncke PMTUD
2002, Cisco Systems, Inc. All rights reserved.

62

GRE+IPSec Encapsulation with Fragmentation

Default behaviour: MTU of GRE tunnel is set to 1476 by IOS
External IP header
FO=0,MF=1,DF=0,ID=1
20 bytes

Original IP datagram (total length = 1500) Original IP header

FO=0,MF=0,DF=0,ID=x
20 bytes

IP payload
1480 bytes

2 GRE packets => 2 IPSec packets => 3 IP packets ESP Original IP header GRE IP payload FO=0,MF=1,DF=0,ID=x Header
16 bytes 4 bytes 20 bytes 1440 bytes

After ESP, 1st GRE was too large The 1st ESP packet is fragmented

External IP header IP payload ESP FO=1480,MF=0,DF=0,ID=1 Trailer

20 bytes 16 bytes 8 bytes

ESP ESP Original IP header GRE IP payload FO=0,MF=0,DF=0,ID=2 Header FO=1456,MF=0,ID=x Trailer External IP header
20 bytes
Evyncke PMTUD

16 bytes 4 bytes

20 bytes

24 bytes

8 bytes
63

2002, Cisco Systems, Inc. All rights reserved.

GRE+IPSec Encapsulation with Fragmentation

MTU of GRE tunnel is set to 1500 in configuration (to avoid fragmentation by GRE)
Original IP datagram (total length = 1500)

Original IP header
FO=0,MF=0,DF=0,ID=x
20 bytes

IP payload
1480 bytes

GRE + IPSec packets (one fragmented GRE + IPSec packet) External IP header
FO=0,MF=1,DF=0,ID=1
20 bytes

IPSec GRE
16 bytes 4 bytes

Original IP header
FO=0,MF=1,DF=0,ID=x
20 bytes

IP payload
1440 bytes

External IP header
FO=1480,MF=0,ID=1
20 bytes

IP payload
40 bytes

IPSec
4 bytes

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

64

GRE+IPSec Encapsulation with Fragmentation

MTU of GRE tunnel is set to 1452 in configuration (to avoid fragmentation by IPSec)
Original IP datagram (total length = 1500)

Original IP header
FO=0,MF=0,DF=0,ID=x
20 bytes

IP payload
1480 bytes

GRE + IPSec packets (Original IP fragmented) External IP header

FO=0,MF=0,DF=0,ID=1
20 bytes

IPSec GRE
16 bytes 4 bytes

Original IP header
FO=0,MF=1,DF=0,ID=x
20 bytes

IP payload
1432 bytes

IPSec
8 bytes

External IP header
FO=0,MF=0,DF=0,ID=2
20 bytes

IPSec GRE

Original IP header
FO=1432,MF=0,ID=x
20 bytes

IP payload
48 bytes

IPSec
8 bytes

16 bytes 4 bytes

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

65

Tactics for GRE+IPSec & Fragmentation

If ICMP are filtered, set ip mtu 1500 on the GRE tunnel
Normal large IP packet
Entrance of tunnel default GRE ip mtu 1500 GRE ip mtu set to lowest GRE tunnel mtu-pathdiscovery 3 fragments 2 fragments In the tunnel path Fragment the fragment Fragment the fragment No fragment Fragment the fragment

IP packet with DF=1

Entrance of tunnel Adjust senders MTU Fragment GRE packet Adjust senders MTU Adjust senders MTU In the tunnel path Fragment the fragment Fragment the fragment No fragment Adjust GRE + IPSec MTU

2 fragments 3 fragments

Evyncke PMTUD

2002, Cisco Systems, Inc. All rights reserved.

66

67