COM:DTU

Course 34355, Routing in Data Networks

Lesson 10 : 19/4-2015 9.30-11.30
Location: Building 324, room 030

Router Design and

Functionality
Litterature :
IP Router Architectures by James Aweya,
Nortel Networks

Background: Whitepaper by Juniper

Supporting Differentiated Service Classes:
Queue Scheduling Disiplines
 Shortly who am I
COM:DTU
Per Flemming Hansen, MSC.EE 1986 from DTU stand-in for Michael S. Berger
pfh@zeuxion.com

I was the teacher of this course in 2007 and 2008, before Michael S Berger.

All years since 1986, I have been working in different company development
departments, participating in development of different packet switch and router
designs-

Dataco, Intel, Tellabs

In Dataco we started doing software based IP routers back in 1989, before Cisco existed.
Forwarding in software, and routing protocols: RIP, OSPF, IS-IS.
Later ASIC development, providing Ethernet/IP stackable switches -> Acquired by Intel 1997.

Technical Co-Founder of Tpack 2001, a Danish venture funded design company, later
sold to Altera in 2008 , today Intel Denmark.
Developed virtual Ethernet/MPLS packet switch for the WAN transition toword packet
switching, included WAN `technologies such as SDH/SONET/VCAT/LCAS and OTN.

Co-Founder of Zeuxion 2007, a small Danish development company doing design

service, mainly for international tier1 customers around the world
packet switch designs, 400 Gbit/s OTN/Ethernet, service OAM, 1588v2-v3
timestamping, mobile integration, FEC, encryption, virtualisation, G5.
Awarded Danish gazelle company in 2012 & 2013.
 Overview
COM:DTU
Until now you have been through
Basic introduction to IP routing & IP networks
Basic routing protocols
RIP, OSPF, BGP4
Multicast Routing Protocols
QoS & Constraint based Routing/Forwarding
Today is about IP routing Architectures
How to build Routers and Packet switch designs.
Focus on Forwarding plane
COM:DTU Existing IP router examples : with very
different purpose

Huawei NE5000E-
Cisco CRS-16 16
12.8 Tbit/s X Terabits
Tellabs/Corian (1 Tbit/s per slot )
t
8840
Alcatel 120 Gbit/s
8000/8600
32 Gbit/s
Wireles Home Router
few 100 Mbit/s

RIPv2 OSPF, BGP4 and many

others
COM:DTU Agenda

Basic Router Functions

Typical IP Packet switch designs
Examples
Evolution of How to build faster and
bigger Router and packet switch
Architectures
Switch Fabrics
COM:DTU

Basic
Router
Functions
COM:DTU IP Network Router Concept

Routing in the IP Network layer of data communication

Control plane (sw)

Routing Protocol
IP Router
Collect Network topology and
calculates forwardTable
Network layer
(IP)
Dynam
link link ic
Netwo
rk
physical physical Topolo
gy Next Hop
ForwardTabl
e
Forward plane (often fast hw)
doing packet forward
COM:DTU
Routing in the IP Network layer of data communication

Upper Upper
Applicatio Applicatio
n layers n layers

Transport Transport
layer (TCP) layer (TCP)
Network Network Network Network
layer (IP) layer (IP) layer (IP) layer (IP)
link lin lin lin lin link
k k k k
physical physic physic physic physic physical
al al al al
 The life of an IP packet through a network from PC1 to
COM:DTU PC2
IP part does not change (apart from TTL and therefore
checksum)
IP Packet
IP1
IP1, IP2 IP Payload
PC1
MAC1 MAC2, IP1, IP2 IP Payload CRC
ETH MAC1t=0x800

Switch
Ethernet switching
ETH

MAC2 Switch MAC2, IP1, IP2 IP Payload CRC

MAC1t=0x800
IP1, IP2 IP Payload
IP IP
Router Router IP Routing
MAC4, IP1, IP2 IP Payload
MAC3 MAC3t=0x800
CRC

ETH Ethernet switching

MAC4, IP1, IP2 IP Payload
MAC4 Switch MAC3t=0x800
CRC

IP1, IP2 IP Payload

IP IP
IP Routing
Router Router
MAC6, IP1, IP2 IP Payload CRC
MAC5 MAC5t=0x800

ETH Ethernet switching

MAC6 Switch MAC6, IP1, IP2 IP Payload CRC

MAC5t=0x800
IP1, IP2 IP Payload
PC2
IP2
COM:DTU

Next hop IP Addr

COM:DTU
COM:DTU

- Route table lookup, packet classification

(SNMP, router config DHCP, BOOTP etc

(RIP, OSPF, BGP, etc

- Exceptions: Source routing option, MTU fragmentation, reassembly

ICMP
COM:DTU
COM:DTU
COM:DTU
COM:DTU Agenda

Basic Router Functions

Typical IP Packet switch designs
Technology Examples
Evolution of How to build faster and
Bigger Router and packet switch
Architectures
Switch Fabrics
 Typical Concept example of N ports IP router
COM:DTU
OSPF, RIP, BGP-4, SNMP, ARP
Very critical
CPU for Memory for due to
Control plane temporay performance
Link
storage
Addr
Matc
Packet Update
h
Packet buffer
(l2header
Management
change etc)
Interfac
Interfac
e to Classificaition e from
output &
Output Output Output input
Ports Queue Rando Policing Input Policing
Schedu Ports
ler & s m IP
Shaper Early Address
Drop Forward
Memory for policing, Lookup
Memory for IP addr
RED, output queues, table (search tree)
shapers

100 Gbit/s gives worts case approx 160 Million minimum packets per second (ETH).
With e.g. a 200 MHz clock, that is a packet address lookup every 1.25 clock.
 Typical Concept example of N ports IP
COM:DTU router
OSPF, RIP, BGP-4, SNMP, ARP
CPU for Memory for
Control plane temporay
storage Link
Addr
Matc
h
Packet Packet buffer
Update Management

Interfac
Interfac
e to Classificaition
& e from
output Output Output Output
Input Policing input
Ports Schedu Queue Rando Policing
Ports
ler & s m IP
Shaper Early Address
Drop Forward
Memory for policing, Lookup
Memory for IP addr
RED, output queues, table (search tree)
shapers
 Typical Concept Example of an N port IP router To
COM:DTU CPU
OSPF, RIP, BGP-4, SNMP, ARP
CPU for Memory for
Control plane temporay
storage Link
Addr
Matc
h
Packet Packet buffer
Update Management

Interfac
Interfac
e to Classificaition
& e from
output Output Output Output
Input Policing input
Ports Schedu Queue Rando Policing
Ports
ler & s m IP
Shaper Early Address
Drop Forward
Memory for policing, Lookup
Memory for IP addr
RED, output queues, table (search tree)
shapers
 Typical Concept Example of an N port IP router
COM:DTU
OSPF, RIP, BGP-4, SNMP, ARP
CPU for Memory for
Control plane temporay
storage Link
Addr
Matc
h
Packet Packet buffer
Update Management

Interfac
Interfac
e to Classificaition
& e from
output Output Output Output
Input Policing input
Ports Schedu Queue Rando Policing
Ports
ler & s m IP
Shaper Early Address
Drop Forward
Memory for policing, Lookup
Memory for IP addr
RED, output queues, table (search tree)
shapers
COM:DTU
 QOS:
COM:DTU
OSPF, RIP, BGP-4, SNMP,
ARP CPU for
Memory for
Control plane temporay
storage Link
Addr
Matc
Packet Update
h
Packet buffer
(l2header
Management
change etc)
Interfac
Interfac
e to Classificaition e from
output &
Output Output Output input
Ports Queue Rando Policing Input Policing
Schedu Ports
ler & s m IP
Shaper Early Address
Drop Forward
Memory for policing, Lookup
Memory for IP addr
RED, output queues, table (search tree)
shapers
COM:DTUExamples of QoS implementation: Priority Queuing

Source: Juniper White paper

http://users.jyu.fi/~timoh/kurssit/verkot/scheduling.pd
 Examples of QoS implementation: Weighted Round Robin
COM:DTU Queuing (packet num oritented)

Source: Juniper White paper

 Examples of QoS implementation:
COM:DTU Deficit Weighted Round Robin Queuing (byte num oriented)

Source: Juniper White paper

 Example of Hierarchical Scheduler:
COM:DTU Combined Prio Queues with shapers and DWRR

Source: Altera/Intel Denmark

COM:DTU Agenda

Basic Router Functions

Typical IP Packet switch designs
Technology Examples
Evolution of How to build faster and
bigger Router and packet switch
Architectures
Switch Fabrics
COM:DTU
FPGA Technologies:
Flexible, and can be reprogrammed in contrast to ASICs.
Has changed a lot in size and speed the last 10-15 years:
2008 technology (90nm -> 65 nm), 2015 technology (14 nm)
Xilinx and Altera fight every year to be the biggest FPGA vendor
and it change.
2008 Altera in front, with first technology target
100Gbit/s linecard card application in a single FPGA.
But since 2009-2010 Xilinx in front, improving with 28
Gbit/s serdes
Today both ready with technlogi target for 400 Gbit/s
linecard applications in a single FPGA.
Intel has now acquired Altera.

Stock pricese has gone up the last few weeks.

ASIC Technologies: (can not be reprogrammed)

(1997 technologi: ~350 nm) : typical target 32*100 Mbit/s packet switch
designs
2015 technlogi, today typical one generation behind FPGA, but faster core
clock rates possible, typical in the range > 400 MHz.
Internal development in the biggest companies: Cisco, Juniper,
 What is possible in FPGAs based solutions ( 2008)
COM:DTU
~2008: 90nm, 65nm Next future: 45nm technology:
technology
internal core clock in FPGA
20 Gbit/s packet switch FPGA chips, :
chips, internal core clock 150 internal core clock 250-300 MHz max
200 MHz max
Note: core clock is not the same as
possible higher interface clocks

Note: ASIC can go higher on internal

core clock
http://www.altera.com/products/devices/stratix2gx/s2
gx-index.jsp
Typically build in
10/100/1000
Ethernet MAC/PHYs

http://www.xilinx.com/products/silicon_solutions/fpgas/virtex/virtex
5/index.htm
 http://www.micron.com/products/dram/rldram/partlist
COM:DTU
Micron highspeed RLDRAMs for packet buffering . E.g. 533 MHz 576
Mbit device
Possible packet speed with 5 devices ( ~180*533 MHz = 95 Gbit/s
-- in/out)

RLDRAM as packet
buffer

FPGA
RLDRA
M
180 bit @ 533
95 Gbit/s packet switch
MHz

With a minimum packet of e.g. 64

bytes this gives a needed IP addr
lookup rate of 184 M
 What is possible in brand newests FPGAs based
COM:DTU solutions ( 2015)
Today: 16->14 nm technology

Altera Stratix 10: Possible to obtain 1.3 Tbps bandwidth

for parallel memmory interfaces with support for DDR4
at 3200 Mbps
Can be used to provide e.g. 400 Gbit/s
Ethernet interfaces, and has support for
up to 56 Gbps tranceivers

http://www.altera.com/products/devices/stratix2gx/s2
gx-index.jsp

http://www.xilinx.com/products/silicon-devices/fpga/virtex-
ultrascale-plus.html
 http://www.micron.com/products/dram/rldram/partlist
COM:DTU
Micron highspeed DDR4 RAM e.g. 1600/3200 per pin speed:
http://www.micron.com/products/dram/ddr4-sdram
 Another Example with A NPU and
COM:DTU FPGA
 Xelerated (Sweden): A Network Processor Unit (NPU)
COM:DTU ASIC

NPU: Target for up to 40 Gbit/s

line cards
NPU is a fast software
programable ASIC

http://www.xelerated.com/templates/page.aspx?page_id=177
 40 Gbit/s Line card Example
COM:DTU
Xelerated NPUs combined with Dune Traffic manager to
backplane

http://www.xelerated.com/templates/page.aspx?
 Combined Ethernet switch & IP WAN
COM:DTU router

2 Mbit/s WAN internet port

E.g. ADSL

IP Router (CPU)

Ethernet Packet Switch (Chip)

16 x 10/100/1000 Ethernet ports

 Such a product type can be build with e.g.
COM:DTU Microsemi/Vitesse 5 port Gigabit Ethernet switch
ASIC (or similar for 100 Mbit/s ports)

http://www.vitesse.com/products/download.php?
fid=269&number=VSC7385
 Combined VLAN Ethernet switch & IP
COM:DTU WAN router

VLAN C Ethernet
Switch
IP
VLAN B Ethernet Route
Switch r

VLAN A Ethernet
Switch

16 x 10/100/1000 Ethernet ports

 Combined VLAN Ethernet switch & IP
COM:DTU WAN router

VLAN C Ethernet
Switch
IP
VLAN B Ethernet Route
Switch r

VLAN A Ethernet
Switch

16 x 10/100/1000 Ethernet ports

 Combined VLAN Ethernet switch & IP
COM:DTU WAN router

VLAN C Ethernet
Switch
IP
VLAN B Ethernet Route
Switch r

VLAN A Ethernet
Switch

16 x 10/100/1000 Ethernet ports

 Combined VLAN Ethernet switch & IP
COM:DTU WAN router

VLAN C Ethernet
Switch
IP
VLAN B Ethernet Route
Switch r

VLAN A Ethernet
Switch

16 x 10/100/1000 Ethernet ports

 Combined VLAN Ethernet switch & IP
COM:DTU WAN router

VLAN C Ethernet
Switch
IP
VLAN B Ethernet Route
Switch r

VLAN A Ethernet
Switch

16 x 10/100/1000 Ethernet ports

Routing between VLAN A and B

 Combined VLAN Ethernet switch & IP
COM:DTU WAN router

VLAN C Ethernet
Switch
IP
VLAN B Ethernet Route
Switch r

VLAN A Ethernet
Switch

16 x 10/100/1000 Ethernet ports

Routing between VLAN A and C

 Broadcom is a company doing these LAN
COM:DTU chips

http://broadcom.com/collateral/pb/56224-PB00-
R.pdf
 Question: What do I need to change in this
COM:DTU architecture, if I want to make an Ethernet switch
instead of a IP router ?
CPU for Memory for
Control plane temporay
storage Link
Addr
Matc
Packet Update
h
Packet buffer
(l2header
Management
change etc)
Interfac
Interfac
e to Classification e from
output &
Output Output Output input
Ports Queue Rando Policing Input Policing
Schedu Ports
ler & s m IP
Shaper Early Address
Drop Forward
Memory for policing, Lookup
Memory for IP addr
RED, output queues, table (search tree)
shapers
COM:DTU

Other
Examples
of
IP Routers
COM:DTU Existing IP router examples : with very
different purpose

Huawei NE5000E-
Cisco CRS-16 16
12.8 Tbit/s X Terabits
Tellabs 8840 (1 Tbit/s per slot )
120 Gbit/s

Alcatel
8000/8600
32 Gbit/s
Wireles Home Router
few 100 Mbit/s

RIPv2 OSPF, BGP4 and many

others
COM:DTU From 2008
COM:DTU From 2008
COM:DTU But 2015

http://www.cisco.com/c/en/us/products/routers/carrier-routing-
system/models-comparison.html

Today 100 Gbit/s interface cards also available.

COM:DTU

The evolution of
how to build
Faster & Bigger
Router
Architectures
COM:DTU
COM:DTU First CPU based Router
architectures

Bandwidth limitation: the shared Bus and processing power

COM:DTU

Route Cache nice speedup for Company router, but not well target for Core
COM:DTU
COM:DTU
COM:DTU

Decentralized router, each line card is in effect a full forward router,

Control Plane still central
COM:DTU

Switch
Fabrics
(to e.g. connect between input and output
boards
in rack backplanes)
COM:DTU What is a switch fabric
Cell based to limit delay due to interleave scheduling between different input ports
COM:DTU
 Switch Fabric
COM:DTU
Responsible for routing packets from input modules to output
modules
Complicated by other requirements such as
multicasting
fault tolerance/redundancy
delay priorities
Important considerations:
Through put, should ideally be non blocking
Minimize Packet loss
Minimize Packet delays
Keep Amount of buffering low
Complexity as simple as possible

Different designs exist:

shared medium
shared memory
distributed output buffered
space division (crossbar)
COM:DTU

Output queued by nature,

which optimize
throughput, delay, and
better delay jittter control
Dedicated non shared
output buffers
Broadcast easy
Single bus N*S
COM:DTU

Output queued nature

Shared Dual Port RAM
Complex with
scheduler, multicast,
single point of failure
Best suited for small
capacity systems
COM:DTU

Output queued by
nature
Independent paths
exist between the N*N
pairs of input,output
Multicast easy
Queue and filter can
operate at ports
speed.
But N*N is the limit
COM:DTU

Space division
The other output queued approach
has the limit that output queue
must match fastest input port.
For terabits output queued is
difficult, so input buffered
crossbars due to low cost and
scalability, but with HOL. Speedup
helps
Combine VOQ (input+output)
Multicast not that easy
COM:DTU
COM:DTU
 Example: Enigmasemi 360 Gbit/s non-blocking
COM:DTU switch core. 2008
Can be combined to provide

http://enigmasemi.com/products/switches.html