DistributedOverlayVirtualEthernet (DOVE)Networks

RenatoRecio IBMFellow& SystemNetworkingCTO

2012 IBM Corporation

EarlyEthernetCampusEvolution
Campus WAN
Core Layer

Layer3

Aggregation Layer Access Layer

Inthebeginning,Ethernetwasusedto interconnectstations(e.g.dumbterminals), initiallythroughrepeater&hubtopologies, eventuallythroughswitchedtopologies. Ethernetcampusesevolvedintoa structurednetworktypicallydivided intoaCore,Service(e.g.firewall), AggregationandAccessLayer.

TrafficpatternismostlyNorthSouth (directedoutsidecampusvs peerpeer). Toavoidspanningtreeproblems,campus networkstypicallyaredividedataccess.

Layer2

95%

5%

2012 IBM Corporation

EthernetintheDataCenter
Data Center WAN
Core Layer

Aggregation Layer

< 25%

Thesamestructurednetworktopologies wereusedinthedatacenter,but
TrafficpatternismostlyEastWest (e.g.ApplicationtoDatabasetier). Largelayer2domainsneededfor clusteringandVirtualMachinemobility.

Layer2

Access Layer

Layer2

> 75%
SAN

PartlyduetoEthernetlimitations (e.g.lackofflowcontrol),thedatacenter usedadditionalnetworks,suchas

FibreChannelStorageAreaNetworks(SAN) InfiniBandclusternetworks.

2012 IBM Corporation

TraditionalDataCenterNetworkIssues
Discrete& Decoupled
Discrete componentsandpieceparts Multiplemanagersand managementdomains BoxlevelpointServices Dynamicworkload managementcomplexity Multitenancycomplications SLAs&securityareerrorprone Toomanynetworktypes, withtoomanynodes&tiers Inefficient switching Expensive networkresources

Manual& Painful

Limited Scale

2012 IBM Corporation

Clients are looking for smarter Data Center Infrastructure that solves these issues.

2012 IBM Corporation

SmarterDataCenterInfrastructure
Expandable IntegratedSystem

Integrated

Simple,consolidatedmanagement SoftwareDrivenNetworkstack

WorkloadAwareNetworking

Automated Dynamicprovisioning
Wireonce fabric.Period.

Converged network

Optimized

Single,flatfabric Growasyouneedarchitecture

2012 IBM Corporation

Optimized

TraditionalArcaneDCN MeshAhead

Multitieredtreetopologies Highoversubscription Expensive,highbandwidthuplinks Smalllayer2fabric Robustnessofhighertierproducts hasbeenaconcern

Mesh,Clos,Jellyfishtopologies OversubscriptiononlytoWAN/core HighcrosssectionalB/W (cheapTORB/W) Layer2scalingoptions(morenext) Robust,HAtopologies

2012 IBM Corporation

Optimized

StandardFabricTechnologyOptions
OpenFlow Controllers OpenFlowSwitches

OpenFlow Layer3
Largelayer2 Establishedtechnology Distributedcontrolplane Standardsbased Largescalability Distributedcontrolplane HAwithfastconvergence Largescalability Emergingtechnology HAwithfastconvergence (someproprietary) Smalllayer2without Singledisjointmultipath DOVEnetwork fabricmayneednewRFC Manydevicestomanage
2012 IBM Corporation

ECMP

TRILL

TRILL

Largelayer2 Distributedcontrolplane Largescalability HAwithfastconvergence Enablesnetwork functionsdeliveredas Services e.g.disjoint multipathing (morelater) Emergingtechnology Clientacceptancebarrier

Optimized

TRILLFabric
Migrationscalability

Keyfabricrequirements
Storage&clusterrequire fullpathredundancy and efficientmultipathing. Largelayer2forVMs.

VM

TRILL Fabric 1 Cores

Single TRILL Fabric

TRILL Fabric 2 Cores

Completelyredundant TRILLfabricsmeetthese requirements,butwith administrationburden.

2xSANconfiguration& maintenance

DualTRILLfabrics areemergingtoday.
Sharedmultipath Disjointmultipath
9

2012 IBM Corporation

SingleTRILLfabric,with disjointmultipathing simplifiesadministration.

Optimized

OpenFlowBasedFabricOverview
EachswitchhasLayer2 forwardingturnedoff. Eachswitchconnectsto OpenFlow Controller (OFC). OFCdiscoversswitches andswitchadjacencies.
OpenFlow

OpenFlow canalsobeusedtocreate adisjointmultipathing fabric.

OpenFlow Controller (Cluster)

OFCcomputessharedor disjointphysicalpaths andconfiguresswitch forwardingtables.

Sharedmultipath Disjointmultipath
10

2012 IBM Corporation

Manual& Painful

Virtualizationincreased networkmanagementcomplexity
Physical Network with vSwitches
App Server Database Server vSwitch Server vSwitch Server vSwitch Server

Before Virtualization
Web Server

Static workloads ran on bare-metal OS Each workload had network state associated with it. Physical network was static & simple (configured once)
11

Server virtualization = dynamic workloads VMs network state resides in vSwitch/DCN Physical network is dynamic and more complex (VMs come up dynamically & move)

2012 IBM Corporation

http://dilbert.com/terms/

Automated

IBMVMready andDVS5000V
VMreadyPhase3 withDVS5000V VMMigration

VMreadyPhase2
ShippingsinceNov2008

VMMigration

vSwitch Server

vSwitch Server

vSwitch Server

5000V
Server

5000V
Server

5000V
Server

NMotion FollowsVM

NMotion PrecedesVM

Closedloopverification withvCenter

1. PerVMswitchinginHW 2. HypervisorVendorAgnostic 3. ManagementPlatformIntegration withVMwarevCenter

12

4. IEEE802.1Qbgstandardbased 5. IBMDistributedVirtualSwitch5000V forVMware 6. NetworkstatemigratesaheadofVM

2012 IBM Corporation

Automated
IBM BC-H IBM Rack Server

VM VM VM VM
DVS 5000v DVS 5000v DVS 5000v

VM
DVS 5000v

VM
DVS 5000v

IBM System Networking OS

feature set on a distributed, virtual switch for VMware

Seamless integration with

VMware vCenter

Standards (Qbg) based network

virtualization coordination between Hypervisor & physical switch

Optimizes East-West traffic

Virtual Machine network state migration with Qbg Virtual Machine network state migration with Qbg Communication path between Virtual Machines

between VMs in a single server (VEB) & VMs within a rack/chassis (VEPA)

Administration simplicity (VEPA)

13

Automated 1000 100 10 1 2006

NetworkVirtualizationTrends
VirtualMachinesper2SocketServer
(approximately10xevery10years)

2008
Infrastructure Groupware

2010
Database Web

2012
Email Application

2014
Terminal Server

2016

NumberofVMspersocketisrapidlygrowing(10xevery10years).
IncreasesamountofVMVMtrafficinEnterpriseDataCenters (e.g.coresidentWeb,Application&Database). VMgrowthincreasesnetworkcomplexityassociatedwithcreating/migrating: layer2(VLANs,ACLs)&layer3(e.g.Firewall,IPS)attributes.
14

2012 IBM Corporation

Automated

HypervisorNetworkVirtualization TechnologyTrend
DOVE
Server Server Service VM1 Server

Service VM2 Server

DC 1

DC 2

Layer2vSwitch features,plus: 1. Layer3DistributedOverlay VirtualEthernet(DOVE) 2. Simpleconfigureoncenetwork (physicalnetworkdoesnthaveto beconfiguredperVM). 3. Decouplesvirtualfromphysical 4. Multitenantaware 5. Enablescrosssubnet virtualapplianceservices (e.g.Firewall,IPS)

15

2012 IBM Corporation

Automated

DOVETechnology VXLANbasedEncapsulationExample
Original Packet
Inner MAC Inner IP Payload

Encapsulation
Outer MAC Outer IP UDP EP Header Inner MAC Inner IP Payload

Version

I R R R Domain ID

Reserved Reserved

Encapsulation Protocol (EP) Header (e.g. VXLAN based) (VXLAN extension in Yellow necessary IETF version field)

16

2012 IBM Corporation

Automated

OverviewofDOVETechnologyElements
DOVES

DOVES

Physical network

DOVE Controller

Overlay Network

DOVEController
Performsmanagement&aportionofcontrolplanefunctions acrossDOVESwitches

DOVESwitches(DOVES)
Provideslayer2overUDPoverlay(e.g.basedonOTV/VXLAN) Performsdataandsomecontrolplanefunctions RunsinHypervisorvSwitch orgateways ProvidesinterfacesforVirtualAppliancestopluginto (Analogoustoappliancelinecardsonamodularswitch)
17

2012 IBM Corporation

Integrated

DOVETechnology+Multipathing
DOVE Gateways Software Defined Networking Stack
(more next)

Standards based multi-pathing

..
DOVE controls overlay network forwarding

..

DOVEnetworksimplifiesvirtualmachinenetwork
EnablesmultitenancyallthewaytotheVM EnablessingleMACAddressperphysicalserver(2forHA) SignificantlyreducessizeofphysicalnetworkTCAM&ACLtables Increaseslayer2scalewithinDataCenterandacrossDataCenters, bydecouplingVMslayer2fromphysicalnetwork Qbg automateslayer2provisioning,DOVEautomateslayer37provisioning

Standardsbasedmultipathed physicalnetwork
18

2012 IBM Corporation

Integrated

SystemNetworkingElementManager
Perform Efficient Firmware or Configuration Updates to Multiple Switches

Engineer

Operate

Automate VM network resident port profiles and converged fabric Quality of Service

Plan

Performance trend & root-cause analysis, fault management, ..

2012 IBM Corporation

19

Integrated

SoftwareDefinedNetworking Technologies
NetworkController
Multi-tenant Services SAN Services

Path Services MultiElement Manager

Networkfunctions deliveredasservices

NetworkAPIs NetworkOperatingSystem
NativeSwitch (L2/3)Driver DOVE Driver OpenFlow Driver

MultitenantVMsecurity Virtualizedloadbalancing

NetworkAPIsprovides anabstractinterface intounderlyingcontroller

Distributes,configures&controls statebetweenservices& controllers Providesmultipleabstractviews

Software HW&embeddedSW

ControlPlane

NetworkOperatingSystem drivessetofdevices
5KV 5KV 5KV 5KV

Physicaldevices(e.g.TOR) Virtualdevices(e.g.DVS5000v)

20

2012 IBM Corporation

SummaryofTechnologyTrends
CAPEX OPEX ExpandableIntegratedSystem Technologies SDNStack (OpenFlow,DOVE,Services) IntegratedManagement

Integrated SystemNetworkElement
Manager SoftwareDefinedNetwork VirtualMachinenetwork stateautomation Multitenantaware NetworkHypervisor

Automated

vSwitch withQbg DOVESwitch

Singlemanagedstackedswitch

Optimized Converged& Flexiblefabric

OptimizedTraffic

Multipathing Converged EnhancedEthernet

21

2012 IBM Corporation

Thank You !

22

2012 IBM Corporation

Backups
Automatingcoordinationoflayer2state IBMDVS5000voverview Examplesofsomeofthevalues associatedwithDOVETechnology
Multitenancy Efficiency

SummaryofTechnologyTrends

23

2012 IBM Corporation

Automated Virtualization

AutomatingLayer2Stateusing IEEE802.1Qbg(VMCreation) 6 VM is brought on-line

after VDP completes Push VM & Port Profile info to servers virtualization infrastructure

Physical End Station

Apps VM Apps VM Apps VM Apps VM

System Admin VM Manager

vSwitch 4
VSI Discovery and Configuration Protocol (VDP)

2 Query available Port

Profile types & select one Push Manager ID and Address

Network Admin

0
VSI Manager
Database

Switch (a.k.a. Bridge) 5

1 Create set of
Virtual Port Profiles

Retrieve Port Profile Information

L2 net(s)

24

2012 IBM Corporation

Automated Virtualization

IBMDVS5000VCapabilities
IBM DVS 5000v Internal External
IBM

VM-VM Virtual Switching Mode Virtual Switch Provider IBM Flexibility to use a mix of internal Yes and external switching modes Eliminates the need to manage the o No virtual switch

Yes

Standard based IEEE 802.1 de-facto IEEE 802.1 de-facto version 0 VEPA & VDP network virtualization coordination version 0 VEPA & VDP Requires new hardware No Sophisticated switch attributes Yes (e.g. ACLs, QoS) A. Internal Yes Automated Switch migration of port B. External profiles Yes Switches Cross Data Center VM migration
No Yes (External switchs)

N/A
Yes - With external switch that supports Qbg

Yes - Through partners standard based approaches (e.g. MPLS/VPLS)

25

2012 IBM Corporation

Automated Virtualization

ImprovingNetworkingEfficiencyfor ConsolidatedServers
Layer-3 Layer-2 Distributed vSwitch
APP APP HTTP A Virtual Machine

Site

vAppliance 10.0.5.7 10.0.5.4 00:23:45:67:00:04 APP 00:23:45:67:00:14 10.0.5.1 10.0.5.5 00:23:45:67:00:01 APP 00:23:45:67:00:15 10.0.0.42 10.0.3.6 00:23:45:67:00:25 Database 00:23:45:67:00:16 Server

Layer-3

HTTP HTTP Server

10.0.3.41 00:23:45:67:00:23 10.0.3.3 00:23:45:67:00:24

10.0.3.9 00:23:45:67:00:17 10.0.3.8 00:23:45:67:00:18

HTTP HTTP

Appliance (e.g.IPS)

vAppliance

HypervisorvSwitches enableadditionofvirtualappliances(vAppliances), whichprovidesecurecommunicationacrosssubnets(e.g.APPtoDatabasetier).

However,alltrafficmustbesenttoanexternalLayer3switch, whichisinefficientconsideringVM/socketgrowthratesandintegratedservers.

TosolvethisissuerequirescrosssubnetcommunicationsinHypervisorsvSwitch.
26

2012 IBM Corporation

Automated Virtualization

MultiTenantwith OverlappingAddressSpaces
vAppliance

A Virtual Machine

Note, vSwitches and vAppliances are not shown.

Site

HOST

10.0.3.1 00:23:45:67:00:01

Database

Coke Overlay Network

Pepsi Overlay Network

Site

10.0.5.7 00:23:45:67:00:04 APP 10.0.3.1 00:23:45:67:00:01 HTTP 10.0.0.4 00:23:45:67:00:25 Database HTTP Server 10.0.3.42 00:23:45:67:00:01

10.0.5.7 00:23:45:67:00:04 10.0.3.42 00:23:45:67:00:25 10.0.5.1 00:23:45:67:00:01 10.0.5.4 00:23:45:67:00:01

HTTP APP HTTP HTTP

vAppliance

Multitenant,CloudenvironmentsrequiremultipleIPaddressspaces withinthesameserver,withinaDataCenterandacrossDataCenters(seeabove).
Layer3DistributedOverlayVirtualEthernet(DOVE)switchesenable multitenancyallthewayintotheServer/Hypervisor, withoverlappingIPAddressspacesfortheVirtualMachines.
27

2012 IBM Corporation