Vous êtes sur la page 1sur 29

Distributed Logical Router

© 2014 VMware Inc. All rights reserved.


Sections
Section 1
• Introduction to the Distributed Logical Router
Section 2
• Logical Router Deployment
Section 3
• Data Plane Packet Walks

Distributed Logical Router CONFIDENTIAL 2 | 29


Introduction to the
Distributed Logical
Router
Section 1
Distributed Logical Router
Overview

• Routing between virtual networks


without leaving virtual space
V V • Layer 3 data plane distributed in
M M hypervisor
V
• Layer 3 control plane running in a VM
M
• Dynamic routing protocols for route
V discovery and advertisement
V • Simplified deployment using NSX
M
M Manager UI or API
V
M
Scale & Performance

• 1000 Logical Interfaces per distributed


V
logical router instance
V M
V • 1200 distributed logical router
M instances total
V M
• 100 per ESXi host
M
• Line rate performance per hypervisor
V V
V V M
M Use Cases
M M
• Optimize routing and data path in
virtual networks
• Supports single tenant or multi tenant
deployment models
No Distributed Router – Hair Pinning
NSX Services Edge GW

7 VM Packet delivered to the


destination
VM
VXLAN 5002
4
1 VXLAN 5001
Packet delivered
vSphere Distributed Switch to the Gateway
10.20.10.12 Interface for
10.20.10.10
VM on Green Logical Routing
Switch communicates vSphere Host vSphere Host
with VM on Red
Logical Switch

Compute Edge/Mgmt.
Rack 1 Rack

After the Routing decision,


the frame is sent to the
Frame sent over
VM on Red Logical Switch 5 3 Frame delivered
to the destination
VXLAN transport
Network to the 2 6 VTEP
Gateway IP of Green
Logical switch

VXLAN Transport Network

Distributed Logical Router CONFIDENTIAL 5 | 29


Distributed Logical Router – Logical View
Logical Logical Router Instance 1

Web VM App VM

VM VM

VXLAN 5001 VXLAN 5002

Logical Router Instance 2

Web VM App VM

VM VM

VLAN 10 VLAN 20

Distributed Logical Router CONFIDENTIAL 6 | 29


Distributed Logical Router – Physical View
Physical LR Control VM

App VM Control Plane NSX Mgr.


Web VM
VM
VM Mgmt. Plane
VXLAN 5002
VXLAN 5001

vSphere Distributed Switch


Controller
10.20.10.10 Data Plane 10.20.10.12 Cluster
vSphere Host vSphere Host

LR Kernel Module

Host 1 Host 2 Control Plane

Controller Nodes

VXLAN Transport and


Management Network

Distributed Logical Router CONFIDENTIAL 7 | 29


Data Path – Host Components
• Logical Interfaces (LIFs) on Logical Router Instance
– IP addresses are assigned on the LIFs
– Multiple LIFs can be configured on one Logical Router Instance
– The LIF configuration is distributed to every Host
– An ARP table is maintained per LIF

• vMAC is the MAC address of the LIF


– vMAC is same across all the Hosts and it is never seen by the physical
network, only by VMs
– VMs use the vMAC as their default gateway MAC address

• pMAC is the MAC address of the uplink through which traffic flows to
the physical network
– In the case of VLAN LIFs the pMAC is seen by the physical network

Distributed Logical Router CONFIDENTIAL 8 | 29


Control Plane - Components
• Distributed Logical Router Control Plane is provided by a per instance
Logical Router Control VM and the NSX Controller
• Supports Dynamic Routing Protocols
– OSPF Logical Router
Control VM
– BGP

• Communicates with NSX Manager and Controller


Cluster
– NSX Manager sends LIF information to the Control
VM and Controller Cluster
– Control VM sends Routing updates to the Controller
Cluster

• High availability supported through Active-Standby configuration

Distributed Logical Router CONFIDENTIAL 9 | 29


Management, Control and Data Communication
Dynamic routing protocol is
External Network
configured on the logical router 1
instance

192.168.100.3 NSX Edge Controller pushes new logical router


(Acting as next hop router) Configuration including LIFs to
ESXi hosts
2
NSX Mgr
Logical Router
Control VM OSPF/BGP peering is established
192.168.10.1
Control between the NSX Edge and logical
1 router control VM. The Protocol 3
192.168.10.3 address is used for Control
Data 6 Communication
3
4 Control
Learnt routes are pushed to the 4
Controller
192.168.10.2 Controller cluster for distribution
2 Cluster
Logical
Router 5 Controller sends the route updates
to all ESXi hosts 5
172.16.10.1 172.16.20.1
Routing kernel modules on the
Web DB
hosts handle the data path traffic
6
VM VM
172.16.10.10 172.16.20.10

Distributed Logical Router CONFIDENTIAL 10 | 29


Questions

Distributed Logical Router CONFIDENTIAL 11 | 29


Logical Router
Deployment
Section 2
VLAN LIF
• The logical distributed Router supports VLAN backed distributed
portgroups
– First hop routing is handled on the host and traffic is switched to the
appropriate VLAN
– A designated instance is required per VLAN LIF

• A VLAN ID must be defined on distributed portgroup


– VLAN ID of 0 is not supported

• VLAN LIFs can only span one VDS

Distributed Logical Router CONFIDENTIAL 13 | 29


Designated Instance
• The designated instance is the host responsible for resolving ARP on a
VLAN LIF
– There is one designated instance per VLAN LIF
– Any ARP request in the distributed portgroup will be handled by the
designated instance
• The NSX Controller makes the designated instance selection
– The NSX Controller pushes designated instance selection to all other hosts

• When the designated instance fails:


– The NSX Controller elects another host as the designated instances
– Informs the remaining host of the new designated instance

Distributed Logical Router CONFIDENTIAL 14 | 29


VXLAN LIF
• The logical distributed router supports VXLAN backed logical switches
– First hop Routing is handled on the host and traffic is switched to the
appropriate logical switch
• If destination is at another hosts, the Ethernet frame is placed inside a VXLAN
frame and forwarded
– A designated instance is not required

• Only one VXLAN LIF can connect to a logical switch


– Next hop router would be a NSX Edge Services Gateway

• Can span all VDS in the transport zone


• Logical distributed routers perform best with VXLAN LIFs

Distributed Logical Router CONFIDENTIAL 15 | 29


Logical Network Topology – VXLAN LIF

External Network

192.168.100.3
NSX Edge
Services Gateway
Active-Standby

192.168.10.1
VXLAN 5002

192.168.10.2

Distributed Logical
Router Instance 1

172.16.20.1
VXLAN 5001
172.16.20.0/24

VM
172.16.20.10

Distributed Logical Router CONFIDENTIAL 16 | 29


Logical Network Topology – VLAN and VXLAN
LIFs

Logical Router
Instance 1

192.168.10.1 192.168.20.1

VXLAN 5002 VLAN 10

192.168.10.0/24 192.168.20.0/24

VM VM
192.168.10.10 192.168.10.11 192.168.20.11

Distributed Logical Router CONFIDENTIAL 17 | 29


Deployment Models – 1 Tier
• 1 Tier of Routing External
– Distributed for east-west Networks
– Designated instance
for north-south
• Dynamic Routing
to advertise logical networks VLAN VXLAN
OSPF Uplink Uplink
BGP
Logical Distributed Router

LB
Web App DB

Web App DB

Distributed Logical Router CONFIDENTIAL 18 | 29


Deployment Models – 2 Tier
• 2 Tiers of Routing
External
– Distributed for east-west Networks
– Perimeter for north-south

• Dynamic Routing to advertise Dynamic Routing


(OSPF, IS-IS, BGP)
logical networks
Perimeter NSX Edge
Transit Uplink 1 Transit Uplink 3

Dynamic Routing Transit Uplink 2


(OSPF, BGP)

Logical Router Logical Router Logical Router


Instance 1 Instance 2 Instance n

Web Logical Web Logical Web Logical


Switch DB Logical DB Logical
App Logical Switch Switch App Logical Switch DB Logical Switch Switch App Logical Switch
Switch Switch

Distributed Logical Router CONFIDENTIAL 19 | 29


Questions

Distributed Logical Router CONFIDENTIAL 20 | 29


Data Plane Packet Walks
Section 3
Distributed Router Traffic Flow – Same Host
DA: vMAC
SA: MAC1 L2 IP Payload L2 IP Payload
192.168.10.10 Logical Router Control VM
DA: 192.168.10.10
192.168.20.10
4 VM SA: 192.168.20.10 Uplink LIF
VM
MAC2
VXLAN 5002
1 MAC1
VXLAN 5001

vSphere Distributed Switch


LIF1 10.20.10.12 LIF1
10.20.10.10 Internal LIFs
LIF2 vMAC LIF2
vSphere Host LIF1 : 192.168.20.1 vSphere Host
LIF2 – ARP Table
3 VM IP VM MAC
LIF2 : 192.168.10.1
vMAC
192.168.10.1
MAC2
Host 1 2 0
Host 2
Routing Table
Destination
Mask Gateway Connect
Interface
255.255.255.
192.168.10.0 0.0.0.0 Direct
0
255.255.255.
192.168.20.0 0.0.0.0 Direct
0

VXLAN Transport Network

Distributed Logical Router CONFIDENTIAL 22 | 29


Distributed Router Traffic Flow – Different Host
DA: vMAC
SA: MAC1 L2 IP Payload 192.168.10.10 L2 IP Payload
192.168.20.10 DA: MAC2
VM SA: vMAC
VM MAC1
DA: 192.168.10.10 MAC2 5
SA: 192.168.20.10
VXLAN 5002
1 VXLAN 5001

VDS
LIF1 LIF1
10.20.10.10 LIF2 - ARP Table 10.20.10.11
LIF2 vMAC LIF2 vMAC
vSphere Host VM IP VM MAC vSphere Host
2 192.168.10.1
0
MAC2

Host 1 L2 IP Payload Host 2


DA: MAC2
SA: pMAC1

3 4

VXLAN Transport Network


DA: 10.20.10.11
SA: 10.20.10.10
DA: MAC2
5002 SA: vMAC LIF1

L2 IP UDP VXLAN L2 IP Payload


Distributed Logical Router CONFIDENTIAL 23 | 29
Traffic Flow – From External Network (Ingress)
Internal Device on the External Network External
Networks (192.168.100.10) communicates with VM on NSX Edge GW
Network
Green Logical Switch (172.16.20.10)
Uplink
172.16.20.10 192.168.100.3
Packet delivered to
VM the destination .1
192.168.10.0/24 VXLAN 5002 Transit Network 2 1
MAC1
6 VXLAN 5001

VDS1 VDS2
10.20.10.10 LIF2 10.20.10.12
vSphere Host
LIF1 vMAC Edge GW routes the traffic
to the next hop router vSphere Host 3
interface 192.168.10.2
LIF2 : 172.16.20.1 (I)
LIF1 : 192.168.10.2 (U)
Host 1 Host 2 The Packets are
After route lookup, the packet is forwarded to Transit
encapsulated in VXLAN header Network LIF
and sent to the VTEP where VM configured on
172.16.20.10 resides Logical Router
4
5
VXLAN Transport Network

Distributed Logical Router CONFIDENTIAL 24 | 29


Traffic Flow – To External Network (Egress)
Internal External
Networks VM on Green Logical Switch communicates with NSX Edge GW Network
Device on External Network (192.168.100.10)
Uplink 192.168.100.3
172.16.20.10

VM Transit .1
192.168.10.0/24 VXLAN 5002 Network 5 6
MAC1
1 VXLAN 5001

VDS1 VDS2
Edge GW sends
10.20.10.10 Packet delivered to the 10.20.10.12 the packet out on
vSphere Host Gateway 172.16.20.1 vSphere Host external network to
2 the destination
192.168.100.10

Destination Interface Mask Gateway Connect


Host 1 Host 2
192.168.10.
192.168.100.0 /24 GW
1

Routing Decision performed and packet sent to next hop


router 192.168.10.1. This is encapsulated VXLAN packet
4
3
VXLAN Transport Network

Distributed Logical Router CONFIDENTIAL 25 | 29


Traffic Flow – From Physical Network (Ingress)
192.168.10.10 192.168.10.11

VM MAC2 VM MAC3
5 GW : 192.168.10.1
Internal LIF
VXLAN 5002
Uplink LIF
VLAN 10

ARP Table vSphere Distributed Switch


LIF1 10.20.10.12 LIF1
IP MAC 10.20.10.10
LIF2 vMAC LIF2 vMAC
192.168.20.1 pMAC2 vSphere Host vSphere Host

LIF1 : 192.168.20.1 pMAC2


192.168.20.11
LIF2 : 192.168.10.1
Host 1 Host 2

GW : 192.168.20.1 Designated
MAC1 Instance
4 2 3
LIF1

L2 Network VXLAN Transport


VLAN 10
VLAN 100
DA: 192.168.10.10
DA: 192.168.10.10 DA: 10.20.10.10 DA: MAC2
SA: 192.168.20.11
SA: 192.168.20.11 SA: 10.20.10.12 SA: vMAC LIF1
5002
DA: pMAC2
SA: MAC1 L2 IP Payload L2 IP UDP VXLAN L2 IP Payload

Distributed Logical Router CONFIDENTIAL 26 | 29


Traffic Flow – To Physical Network (Egress)
192.168.10.10 DA: 192.168.20.11 192.168.10.11
GW : 192.168.10.1 SA: 192.168.10.10
VM MAC2 VM MAC3
1
DA: vMAC
SA: MAC2 L2 IP Payload
Uplink LIF
VLAN 10 VXLAN 5002
Packet delivered
to the destination vSphere Distributed Switch
LIF1 10.20.10.12 LIF1
10.20.10.10
LIF2 vMAC LIF2 vMAC
vSphere Host vSphere Host
10
2 LIF1 - ARP Table
192.168.20.11
VM IP VM MAC
8
VM IP VM MAC
Host 1 Host 2
192.168.20.1
pMAC1 1
MAC1 pMAC2
MAC1 GW : 192.168.20.1 3 Designated
Instance
LIF1

9 4
Out of band UDP channel is established
with DI for ARP resolution on LIF1
7
6 5 ARP request
ARP request sent out by
the DI
L2 Network VXLAN Transport
VLAN 10
VLAN 100

ARP response DA: MAC1 DA: 192.168.20.11


SA: pMAC1 L2 IP Payload SA: 192.168.10.10

Distributed Logical Router CONFIDENTIAL 27 | 29


Design Considerations
• VLAN LIFs introduce some constraints on network design
– One VDS
– Same VLAN spanning all hosts in the VDS (potentially up to 1000 hosts)
– In the recommended design for network virtualization VLAN span is limited
– VXLAN LIFs don’t have these constraints

• Two LIFs can’t be connected to same logical switch or network


• Routing not supported on Bridged Interfaces
• Distributed logical router provides following benefits
– No hair pinning and optimized handling of east-west traffic
– Supports a large number of LIFs (1000)
– Scalable routing topologies

Distributed Logical Router CONFIDENTIAL 28 | 29


Questions

Distributed Logical Router CONFIDENTIAL 29 | 29