Académique Documents
Professionnel Documents
Culture Documents
NCS5500
Deepdive in the Merchant Silicon High-end SP Routers
Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session
How
1 Find this session in the Cisco Events Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
cs.co/ciscolivebot#BRKSPG-2900
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
What We Hope To Achieve With This Session
• Getting familiar with the NCS 5500 portfolio
• Understand the implementation differences compared to traditional
XR products
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Agenda
• Introduction
• Products Portfolio
• Fixed / Modular Platforms / Optics
• VOQ/FMQ and Life of a Packet
• NCS 5500 Internals
• Memory Structure
• Features: ACL / QoS
• Conclusion
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Introduction
IOS XR Routing Products
Virtual RR/PE/DC
Forwarder
Cisco XR Software
Programmability
Elastic Cost Optimized Ultra-high Density Carrier Grade and Automation
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
NCS…
Vast Product Line
Platform / Series
NCS 520
NCS 540
NCS 560
NCS 1000
NCS 2000
NCS 4000
NCS 4200
NCS 5000
NCS 5500
NCS 6000
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
NCS…
At A Glance
Platform / Series Role
NCS 520 Ethernet Access Device (IOS XE)
NCS 540 Access Router
NCS 560 Aggregation Router
NCS 1000 DCI / IP-DWDM
NCS 2000
Packet Optical
NCS 4000
DWDM / TDM to IP / CEM
NCS 4200
NCS 5000 Top of Rack Router
NCS 5500 Core, Edge, Agg, Peering Router
NCS 6000 Core Router
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
NCS 5500 and NCS 5000
Two Very Different Platforms
• Both based on Merchant Silicon forwarding ASICs and running IOS XR 64-bit
• Still they are very different in nature and in position in networks
• NCS 5500
• High scale routing and features
• Exists in Fixed and Modular form factors (Fabric Engine)
• Hybrid Architecture with Deep Buffers
• NCS 5000
• Lower scale and small buffers
• No Chassis with Fabric Engine
• Cost optimized
• Can be used as a nV Satellite for ASR 9000 and NCS 6000
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
NCS 5500 and NCS 500
Much Closer Platforms
• Both based on same Merchant Silicon ASIC family (DNX)
• A lot of commonalities in the architecture and feature support
• Some difference in scale and features related to specific additional hw parts
• NCS 540
• based on Qumran-AX (lower routing scale)
• NCS 560
• Based on Qumran-MX with OP eTCAM (2nd Generation eTCAM)
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
But What is Merchant?
Really…
Components
Merchant/Commodity, Proprietary, Custom
• Merchant
• Not designed by a system vendor
• Available on the open market to any system vendor or network operator
• Proprietary
• Designed or acquired by a router vendor
• Not available to others
• Custom
• Designed in concert with a specific router in mind
• Usually proprietary but may be merchant with extensions
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Custom and Merchant
Cisco Platforms Internal Components
NCS 6000
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
NCS 5500 Products Family External TCAM
Base and Scale
• Both exist for modular and fixed systems
Buffers
• Base QSFP28
QSFP28
• On-chip FIB and small TCAM TCAM Buffers
QSFP28 Forwarding
QSFP28 ASIC
for ACLs / QoS QSFP28 QSFP28
QSFP28 QSFP28
• Scale (-SE) have increased FIB and ACL QSFP28 Forwarding
ASIC
QSFP28 Optics x 6 FA
• off-chip TCAM QSFP28
QSFP28 Optics x 6 FA
• On both platforms: –SE will support more features with higher scale
• But scale will be different
• ASR9000: different QoS capability (because higher classifier scale)
• NCS5500: different FIB scale (because TCAM is used to store routing information,
not only classifiers)
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
NCS 5500
Basics Concepts on NPU
Simplification is Key
Fewer Components: Cost Optimization and Lower Power Consumption
PLA
PHY EgressQ PSE FabricQ
PHY
Fabric Fabric
ASR9900 Optics NPU FIA
ASIC
Fabric ASIC
ASIC
FIA NPU Optics
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
NCS 5500 Forwarding ASIC
Jericho / Qumran-MX / Jericho+ ASICs
900G 1200G
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
NCS 5500 Forwarding ASIC
Jericho / Qumran-MX / Jericho+ ASICs
• Integrated Forwarding and Fabric Interface Network Interface
LPM
Off-chip TCAM
On-chip Buffer OTM
Buffers
• On-chip resources TCAM PP TM
STAT
FEC
PP TM
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
NCS 5500 Forwarding ASIC
Pipeline?
• Run to Completion: many cores, each does everything for a packet
Network Interface
Fabric Interface
1 2 3 4 5 6 7 8 9
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
NCS 5500 Forwarding ASIC
Pipeline Architecture
Packet
Buffer
Packet Packet
DB DB DB DB DB Buffer DB DB Buffer
Network If
Network If
Fabric If
Fabric If
Fabric
IRPP ITM ITPP ETPP ETM ERPP
Network If
Network If
Fabric If
Fabric If
NPU2
Packet Packet
DB DB DB DB DB Buffer DB DB Buffer
Network If
Network If
Fabric If
Fabric If
Fabric
IRPP ITM ITPP ETPP ETM ERPP
Network If
Network If
Fabric If
Fabric If
NPU2
| /|\
| J E R I C H O N E T W O R K I N T E R F A C E |
\|/ |
+-------------------------------------------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
| NBI |
| RX_TOTAL_BYTE_COUNTER = 0 | TX_TOTAL_BYTE_COUNTER = 4,015 |
| RX_TOTAL_PKT_COUNTER = 0 | TX_TOTAL_PKT_COUNTER = 0 |
| RX_TOTAL_DROPPED_EOPS = 0 | |
+-------------------------------------------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
<SNIP>
+-------------------------------------------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
<SNIP>
+-------------------------------------------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
| | FDA |
| | CELLS_IN_CNT_P1 = 0 | CELLS_OUT_CNT_P1 = 0 |
| | CELLS_IN_CNT_P2 = 22 | CELLS_OUT_CNT_P2 = 20 |
+-------------------------------------------+-------------------------------------------| CELLS_IN_CNT_P3 = 0 | CELLS_OUT_CNT_P3 = 0 |
| IPT | CELLS_IN_TDM_CNT = 0 | CELLS_OUT_TDM_CNT = 0 |
| | CELLS_IN_MESHMC_CNT = 0 | CELLS_OUT_MESHMC_CNT = 0 |
| EGQ_PKT_CNT = 0 --> CELLS_IN_IPT_CNT = 0 | CELLS_OUT_IPT_CNT = 0 |
| ENQ_PKT_CNT = 0 | EGQ_DROP_CNT = 0 |
| FDT_PKT_CNT = 0 | EGQ_MESHMC_DROP_CNT = 0 |
| CRC_ERROR_CNT = 0 | EGQ_TDM_OVF_DROP_CNT = 0 |
| CFG_EVENT_CNT = 0 | |
| CFG_BYTE_CNT = 0 | |
+-------------------------------------------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
| FDT | FDR |
| IPT_DESC_CELL_COUNTER = 0 | P1_CELL_IN_CNT = 0 |
| | P3_CELL_IN_CNT = 0 |
| TRANSMITTED_DATA_CELLS_COUNTER = 0 | CELL_IN_CNT_TOTAL = 22 |
+-------------------------------------------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
| /|\
| J E R I C H O F A B R I C I N T E R F A C E |
\|/ |
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
Fixed Platforms
Naming Rules for Fixed Platforms
NCS 55xy-zzH-(SE)-(S)
S = MACsec
x = 0 Jericho based y = #RU zz = 100G ports
x = A Jericho+ based MODular SE = Scale
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
NCS 5500 Fixed Platforms
NCS 5501-SE
• Single 800 Gbps FA, 4GB packet buffer
• 600 Mpps
Buffers
• No Oversubscription, total interfaces: 800G Forwarding
CPU
ASIC
• 40x 1/10G SFP ports
TCAM
• 4x 40/100G QSFP ports
• Support of Timing and DWDM interfaces
QSFP28
QSFP28
QSFP28
QSFP28
DRAM
SFP+
SFP+
SFP+
SFP+
SFP+
SFP+
40x10G 4x100G
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
NCS 5500 Fixed Platforms
NCS 5501
• Single 800 Gbps forwarding ASIC, 4GB packet buffer
• 720 Mpps
Buffers
• Oversubscribed design, Forwarding
CPU
ASIC
total bandwidth of 1.08 Tbps
• 48x 1/10G SFP ports
• 6x 40/100G QSFP ports
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
DRAM
SFP+
SFP+
SFP+
SFP+
SFP+
• No DWDM support
48x10G 6x100G
• No timing support
Product LEM LPM eTCAM
NCS5501 786k 256k-350k -
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
For Reference
NCS 5501
100Mbps / 1Gbps Limitations
• NCS 5501-SE ports 0/8 to 0/15
• Don’t support 100Mbps copper SFP modules (GLC-T)
• Don’t support auto-neg for 1G optical SFP
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
NCS 5500 Fixed Platforms
NCS 5502-SE Switch Fabric Switch
Buffers
(Typical, SR optics)
CPU
Forwarding
FA
FA
FA
FA
FA
FA
FA
ASIC
TCAM
• 48x 100G QSFP28 (or QSFP+)
LC
QSFP x 6
QSFP x 6
QSFP x 6
QSFP x 6
QSFP x 6
QSFP x 6
QSFP x 6
(Common FA with modular chassis)
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
DRAM
• 600 Mpps per FA
48x100G
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
NCS 5500 Fixed Platforms
NCS 5502 Switch Fabric Switch
Buffers
(Typical, SR optics)
CPU
Forwarding
FA
FA
FA
FA
FA
FA
FA
ASIC
QSFP x 6
QSFP x 6
QSFP x 6
QSFP x 6
QSFP x 6
QSFP x 6
QSFP x 6
external TCAM
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
DRAM
• 8x 600 Gbps Forwarding ASICs
48x100G
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
NCS 5500 Fixed Platforms
NCS 5502 Internal Architecture
18
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
DRAM
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
For Reference
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
NCS-55A1-36H-S / NCS-55A1-36H-SE-S
• 1 Rack Unit Fixed System
• 36x QSFP28 or QSFP+ optics
• NCS-55A1-36H-S / NCS-55A1-36H-SE-S
• Base and Scale (-SE) versions
• Scale extension with eTCAM of 4M+ entries
• 4x 900 Gbps Jericho+ Forwarding ASICs
• 1x 3.6Tbps Fabric ASIC
• 1588 / Sync-E Capable
• MACSEC Capable
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
For Reference
NCS-55A1-36H-S / NCS-55A1-36H-SE-S
• Single Intel Broadwell-DE D1577 CPU
• 8-core @ 1.6GHz
• 32GB RAM, 64GB SSD
• 2 Redundant Power Modules: 2kW AC or DC
• Base system: Typical= 1100W / Max Power= 1450W
• Scale system: Typical= 1300W / Max Power= 1700W
• 3 Redundant (N+1)
• Front to Back Fan Modules
Product LEM LPM eTCAM
• Dimension: 1RU / depth: 30 inches
55A1-36H-S 786k 256k-350k -
55A1-36H-SE-S 786k 256k-350k 4M+
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
NCS-55A1-36H-S / NCS-55A1-36H-SE-S
Internal Architecture
eTCAM
eTCAM
eTCAM
eTCAM
MACsec MACsec MACsec MACsec MACsec MACsec MACsec MACsec MACsec MACsec MACsec MACsec MACsec MACsec MACsec MACsec MACsec MACsec
DRAM
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
NCS-55A1-24H
• 1 Rack Unit Fixed System: 24x QSFP28 ports
• Base version only and no MACSEC capability
• 1588 / Sync-E Capable
• 2x 900 Gbps Forwarding ASICs Product LEM LPM eTCAM
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
For Reference
NCS-55A1-24H
• Single Intel Broadwell-DE D1577 CPU
• 8-core @ 1.6GHz
• 32GB RAM, 128GB SSD
• Dimension: 1RU / Depth: 21 inches
• 2 Redundant Power Modules: AC or DC
• Typical= 600W / Max Power= 800W
• 2 Redundant (N+1) Fan Modules: Front to Back (B2F planned)
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
NCS-55A1-24H
Internal Architecture
48x25G
4x 25G
DRAM
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
• Oversubscription of 12x100G ports on 900G Forwarding ASIC
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
NCS-55A2-MOD-S Series
• 2RU, 11 inches deep (280mm)
• 1x Jericho+ Forwarding ASIC
• 835 Mpps / 900 Gbps (160% max oversubscribed)
• Fixed 40x 1/10G SFP/SFP+ DWDM capable
• 24x 1/10G
• 16x 1/10/25G (MACsec at 10/25G)
• 2x 400G Modular Port Adaptor bays
• Timing 1588/SyncE and MACsec Capable
• 8x Fan Modules (F2B), 2x Power Supply AC/DC (Front)
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
NCS-55A2-MOD-S
Stats FPGA
Forwarding ASIC
CPU
eTCAM
Jericho+
4GB Buffers
2x25G
=50G
10G 8x25G
MACsec MACsec
=200G
DRAM
Up to 400G Up to 400G
SFP28
SFP28
SFP28
SFP28
SFP28
SFP28
SFP28
SFP28
SFP+
SFP+
SFP+
SFP+
SFP+
SFP+ MPA0 MPA1
0/0/1/x 0/0/2/x
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
NCS 55A2-MOD
NCS-55A2-MOD-S
• Base version
• Single Intel Broadwell CPU (6 cores @ 2GHz), 32GB RAM, 128GB SSD
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
NCS 55A2-MOD
NCS-55A2-MOD-HD-S
• Base Hardened version
• GR 3108 Class 2
• Expected temperature range: around -40C to +70C
• Single Intel Broadwell CPU (6 cores @ 2GHz), 32GB RAM, 128GB SSD
• Single Temp Hardened MPA option
• MPA 4x QSFP28 (4x10G / 40G / 100G)
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
NCS 55A2-MOD
NCS-55A2-MOD-SE-S
• Scale version
• Single Intel Broadwell CPU (8 cores @ 2GHz), 32GB RAM, 128GB SSD
• External TCAM and FPGA for statistics (future use)
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
Modular Port Adapters (MPA)
NC55-MPA-12T-S Connector NC55-MPA-2TH-S Connector
Up to 16x25G=400G Up to 16x25G=400G
12 x 10G 4 x 25G 4 x 25G 4 x 25G 4 x 25G
SFP+
SFP+
SFP+
SFP+
SFP+
SFP+
SFP+
SFP+
SFP+
SFP+
SFP+
CFP2-DCO CFP2-DCO
(2x100G) (2x100G)
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
NCS-55A2-MOD Series
Timing Capabilities
• IEEE 1588-2008 PTP support
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
NCS-55A2-MOD Series MACSec MACSec
• Capable on last 16xSFP28 fixed ports except 1GE mode MACSec MACSec
•
2 ports CFP2
MACsec MACsec
OTN, MACSec
SFP+
SFP+
SFP+
SFP+
SFP+
SFP+
SFP+
SFP+
SFP+
SFP+
SFP+
SFP+
Up to 400G Up to 400G
SFP28
SFP28
SFP28
SFP28
SFP28
SFP28
SFP28
SFP28
SFP+
SFP+
SFP+
SFP+
SFP+
SFP+
12 ports SFP+
MPA0 MPA1
0/0/1 0/0/2 MACSec MACSec
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
For Reference
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Orthogonal Design
• No backplane/midplane for data path
• Direct connection between LC to fabric cards at 90 degrees
• Air inlets above and between optics
• Air inlets on RP & power supplies
AIR INLETS
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
NCS 5500 Modular Chassis
Mechanical Layout
Fabric Behind Fans
Line Cards
RP RP
Air Intake
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
For Reference
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
For Reference
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
For Reference
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Switch Fabric Cards
• Cell-based fabric
• FE3600 fabric ASIC
• 6 Fabric Cards per chassis
• Same Switch Fabric Cards for both FA generations
• Single-stage full mesh utilizing
• 1 Fabric ASICs / card for NCS 5504
• 2 Fabric ASICs / card for NCS 5508
• 6 Fabric ASICs / card for NCS 5516
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
Switch Fabric Cards
• Cell-based fabric
• FE3600 fabric ASIC
• 6 Fabric Cards per chassis
• Same Switch Fabric Cards for
both Jericho and Jericho+
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
Switch Fabric Cards
• Number of FE ASIC per Fabric Module depends on the form factor
FE
J FE J+ FE
6x25G 8x25G 3x25G 4x25G
=150G =200G =75G =100G FE
J FE J+
FE
J FE J+
1x25G 1 or 2
x25G
FE
FE
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
NCS 5500 Modular Chassis
36x 100G Line Card Bandwidth Example
Fabric Card 0 Fabric Card 1 Fabric Card 2 Fabric Card 3 Fabric Card 4 Fabric Card 5
6x6x25G=900G 900G 900G 900G 900G 900G
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
QSFP
DRAM
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
NCS 5500 Modular Chassis
Common System Controller and Route Processor
• Route Processor • System Controller
• Ivy Bridge with 24GB RAM • Chassis control and monitoring
• Routing and management tasks • Fan trays / Power supply
• Ethernet Out-of-Band Channel (EOBC)
• Ethernet Protocol Channel (EPC)
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
Modular Line Cards based on Jericho
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
For Reference
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
For Reference
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
NCS 5500
Other Platforms
• Preparing the 400G introduction
• 24x 400G LC
• Higher density for 25G
• 48x 25G + 6x 100G 1RU systems
• 24x 25G + 6x 100G 1RU systems
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
NCS 5500 Optics
NCS 5500 Interfaces
Ethernet Only Platforms
• SFP optics slot: offering 1G or 10G (with SFP+) on the following platforms
• NCS 5501: 40/48 ports
• QSFP optics slot: offering 100G (with QSFP28), 40G (with QSFP+) and
4x 10G (QSFP+ with break-out cables) on the following platforms or LC
• NCS 5502: 48 ports
• Line Cards
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
NCS 5500 Interfaces
Ethernet Only Platforms
• 25GE only supported on J+ Platforms with 4x25G break-out
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
NCS 5500 Positioning
NCS 5500 Position in Network
Multi-dimensional Equation
• The position decision of a platform should be based on:
• Ports types / density requirement for X years
• Scale requirements
• Buffering capability
• Supported features
• Power consumption
• Network OS preference (IOS XR)
• No simple rule of thumb
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
NCS 5500 Position in Network
Think about…
• QoS
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
For Reference
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
For Reference
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
VOQ and Life of a
Unicast Packet
NCS 5500 Architecture
Local Routing
• Local traffic on NCS 5500 series can be routed by the FA without going
through the fabric: lower latency
Optics
Optics
Optics
NCS 5502 Forwarding
Fabric ASIC
NCS 5508 Optics
ASIC
PLA
+ OTN PSE
PHY EgressQ PSE FabricQ PHY
Lookup #1 Lookup #2
Egress to identify
Ingress to identify
Interface, VLAN,
destination LC
adjacency
Fabric Fabric
ASR9900 Optics NPU FIA
ASIC
Fabric ASIC
ASIC
FIA NPU Optics
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
NCS 5500 Architecture
Comparison with Traditional XR Platforms
• Single-lookup architecture at ingress on NCS 5500
• VOQ-only Model
Single lookup in
ingress FA
Relevant info set in
internal headers
Forwarding
NCS 5501 Optics
ASIC
Optics
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
NCS 5500 System Architecture
Three Packet Buffers / Hybrid Model
• Ingress On-chip Buffer: 16MB
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
NCS 5500 System Architecture
Three Packet Buffers / Hybrid Model
• Normal traffic condition (no congestion)
• Packets stored in on-chip buffers only
• That’s the 99.999% of the packets
(totally made-up number to say the vast majority of the time)
Egress
Ingress On Chip Buffer Egress Port Buffer Interface
Net FIA FIA Net without
Interface 16MB 16MB
congestion
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
NCS 5500 System Architecture
Three Packet Buffers / Hybrid Model
• In case of egress queue congestion
• Packets stored in ingress off-chip buffers until they receive permission
Egress
Ingress On Chip Buffer Egress Port Buffer Interface
Net FIA FIA Net with queue
Interface 16MB 16MB
congestion
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
NCS 5500 System Architecture
Three Packet Buffers / Hybrid Model
• Contrary to traditional XR platforms: very short egress buffering
• 4 priorities on the egress port buffer
• High Unicast
• High Multicast Egress
Egress Port Buffer Interface
• Low Unicast FIA 16MB Net with queue
• Low Multicast HP Unicast
congestion
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
NCS 5500 System Architecture
VOQ-Only Architecture (Virtual Output Queues)
• We have 8 queues per attachment point
Egress VOQ
Scheduler
LC7
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
NCS 5500 System Architecture
VOQ-Only Architecture (Virtual Output Queues)
• Every NPU will have a logical (virtual) representation of these egress queue
locally where packets are actually stored in congestion situation VOQ
0/7/0/0.2 VOQ0 0/7/0/x VOQ0
0/7/0/0.2 VOQ1 0/7/0/x VOQ1
0/7/0/0.2 VOQ2 0/7/0/x VOQ2 Fabric Card 0
0/7/0/0.2 VOQ3 0/7/0/x VOQ3
0/7/0/0.2 VOQ4 0/7/0/x VOQ4 Connector
Ingress 0/7/0/0.2 VOQ5 0/7/0/x VOQ5
Net
Interface 0/7/0/0.2 VOQ6
0/7/0/0.2 VOQ7
0/7/0/x VOQ6
0/7/0/x VOQ7
Fabric Card 1
VOQ
Ingress VOQ
Scheduler Fabric Card 2
LC0
0/7/0/0.2 Queue0
0/7/0/0.2 Queue1
0/7/0/0.2 VOQ0 0/7/0/x VOQ0
0/7/0/0.2 VOQ1 0/7/0/x VOQ1 Fabric Card 3 0/7/0/0.2 Queue2
0/7/0/0.2 Queue3
0/7/0/0.2 VOQ2 0/7/0/x VOQ2
0/7/0/0.2 Queue4
0/7/0/0.2 VOQ3 0/7/0/x VOQ3
0/7/0/0.2 Queue5
0/7/0/0.2 VOQ4 0/7/0/x VOQ4
Ingress 0/7/0/0.2 VOQ5 0/7/0/x VOQ5
0/7/0/0.2 Queue6
Net Egress
Net Fabric Card 4 0/7/0/0.2 Queue7
Interface 0/7/0/0.2 VOQ6
0/7/0/0.2 VOQ7
0/7/0/x VOQ6
0/7/0/x VOQ7 Egress Port Queues Interface
VOQ Hu0/7/0/0.2
Ingress VOQ Fabric Card 5 Egress VOQ
Scheduler Scheduler
LC1 LC7
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
NCS 5500 System Architecture
VOQ-Only Architecture (Virtual Output Queues)
• Even for the same NPU 0 on the same line card, the ingress pipe uses this
virtual representation (Local VOQ)
0/7/0/0.2 VOQ0 0/7/0/x VOQ0 0/7/0/0.2 VOQ0 0/7/0/x VOQ0
0/7/0/0.2 VOQ1 0/7/0/x VOQ1 0/7/0/0.2 VOQ1 0/7/0/x VOQ1
0/7/0/0.2 VOQ2 0/7/0/x VOQ2 Fabric Card 0 0/7/0/0.2 VOQ2 0/7/0/x VOQ2
0/7/0/0.2 VOQ3 0/7/0/x VOQ3 0/7/0/0.2 VOQ3 0/7/0/x VOQ3
0/7/0/0.2 VOQ4 0/7/0/x VOQ4 0/7/0/0.2 VOQ4 0/7/0/x VOQ4 Connector
Ingress 0/7/0/0.2 VOQ5 0/7/0/x VOQ5 0/7/0/0.2 VOQ5 0/7/0/x VOQ5
Net
Interface 0/7/0/0.2 VOQ6
0/7/0/0.2 VOQ7
0/7/0/x VOQ6
0/7/0/x VOQ7
Fabric Card 1 0/7/0/0.2 VOQ6
0/7/0/0.2 VOQ7
0/7/0/x VOQ6
0/7/0/x VOQ7
VOQ VOQ
Ingress VOQ
NPU0 Ingress VOQ
Fabric Card 2 Scheduler
Scheduler
LC0
0/7/0/0.2 Queue0
0/7/0/0.2 Queue1
0/7/0/0.2 VOQ0 0/7/0/x VOQ0
0/7/0/0.2 VOQ1 0/7/0/x VOQ1 Fabric Card 3 0/7/0/0.2 Queue2
0/7/0/0.2 Queue3
0/7/0/0.2 VOQ2 0/7/0/x VOQ2
0/7/0/0.2 Queue4
0/7/0/0.2 VOQ3 0/7/0/x VOQ3
0/7/0/0.2 Queue5
0/7/0/0.2 VOQ4 0/7/0/x VOQ4
Ingress 0/7/0/0.2 VOQ5 0/7/0/x VOQ5
0/7/0/0.2 Queue6
Net Egress
Net Fabric Card 4 0/7/0/0.2 Queue7
Interface 0/7/0/0.2 VOQ6
0/7/0/0.2 VOQ7
0/7/0/x VOQ6
0/7/0/x VOQ7 Egress Port Queues Interface
VOQ Hu0/7/0/0.2
-------------------------------------------------------------------
Node ID: 0/0/CPU0
Intf Intf NPU NPU PP Sys VOQ Flow VOQ Port
name handle # core Port Port base base port speed
(hex) type (Gbps)
----------------------------------------------------------------------
Hu0/3/0/5 1800100 0 0 1 1537 1072 10280 remote 100
Hu0/0/0/26 200 4 1 17 273 1424 4136 local 100
Hu0/3/0/6 1800108 1 1 21 1621 1080 1064 remote 100
Hu0/0/0/27 208 4 0 9 265 1432 5416 local 100
Hu0/3/0/7 1800110 1 1 13 1613 1088 2344 remote 100
Hu0/0/0/28 210 4 0 5 261 1440 7208 local 100
Hu0/3/0/8 1800118 1 1 17 1617 1096 4136 remote 100
Hu0/0/0/29 218 4 0 1 257 1448 8488 local 100
Hu0/3/0/9 1800120 1 0 9 1609 1104 5416 remote 100
Hu0/0/0/30 220 5 1 21 341 1456 2344 local 100
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
NCS 5500 Forwarding ASIC Detail
Deep Buffer LPM
PP TM PP TM
• Expansion via off-chip resources LEM
Off-chip
• Deep GDDR5 packet buffers external packet buffers Buffers
On-chip Buffer TCAM OTM
STAT
• In normal conditions PP TM
FEC
PP TM
Fabric Card 3
Fabric Card 5
Fabric Card 3
Queue-Status ?
Ingress VOQ Fabric Card 4 Egress VOQ
Scheduler Scheduler
Fabric Card 5
NO Credit
• Ingress VOQ scheduler polls Egress scheduler (maintaining a local VOQ DB)
• Egress answers with a credit-message (or not, in our example)
• Egress device decides how much traffic can be sent by granting credits to any
ingress requesting Forwarding ASIC
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
NCS 5500 VOQ-Only Architecture
Fabric Card 0
Fabric Card 3
Queue-Status ?
Ingress VOQ Fabric Card 4 Egress VOQ
Scheduler Scheduler
Fabric Card 5
Credit
• Finally, the egress schedule grants the credit for packet transmission
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 91
NCS 5500 VOQ-Only Architecture
Fabric Card 1
Fabric Card 4
Fabric Card 6
• Packet is split in cells and load balanced among the fabric cards
• Cells are transported to the egress line card
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 92
NCS 5500 VOQ-Only Architecture
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 93
NCS 5500 VOQ-Only Architecture
Fabric Card 1
Fabric Card 4
Fabric Card 6
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 94
NCS 5500 VOQ-Only Architecture in NCS 5501
Queue-Status ?
Ingress VOQ Egress VOQ
Scheduler Scheduler
Credit
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 95
NCS 5500 VOQ-Only Architecture in NCS 5501
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 96
FMQ and Life of a
Multicast Packet
Multicast in NCS 5500
• L3 multicast support introduced in IOS XR 6.1.1
• Initially limited to Source Specific Protocols and IPv4 only (IGMPv3 and PIM SSM)
• ASM protocols (RP discovery, etc) and MPLS Multicast added later
• (S,G) information stored in LPM and takes one entry each
• IPv4 key (VRF, S, G)
• IPv6 key (VRF, G)
• MCID / FGID
• Replication performed at two levels
• Fabric level
• egress Forwarding ASIC level
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 98
NCS 5500 System Architecture
Control Plane
• IGMP and PIM joins are punted to RP CPU process (igmp/pim)
• Packets use EPC internal network to reach the process executed on RP LXC
MRIB
RP CPU
or
L2FIB
LC1
Hu0/1/0/0
NIF
NPU-0
Hu0/1/0/5
NIF
Hu0/1/0/7 NPU-1
LC2
Hu0/2/0/3
NIF
NPU-0
Hu0/2/0/4
IGMP/PIM Join BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 99
NCS 5500 System Architecture
Control Plane
• If it’s a new group, the process (MRIB or L2FIB) will allocate a Multicast ID (MCID)
• If a MCID is already allocated, information will be updated based on join/leave
MRIB
RP CPU
or MCID 60414
L2FIB
LC1
Hu0/1/0/0
NIF
NPU-0
Hu0/1/0/5
NIF
Hu0/1/0/7 NPU-1
LC2
Hu0/2/0/3
NIF
NPU-0
Hu0/2/0/4
IGMP/PIM Join BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 100
NCS 5500 System Architecture
Control Plane: Identifying MCID
• MCID is often times referred as FGID internally
• You can find the MCID associated to a (*,G) or (S,G) pair with the following CLI:
(50.41.13.11,232.31.0.12) Ver: 0xef18 RPF nbr: 16.2.4.1 Flags: RPF, FGID: 9155
Up: 04:20:11
Incoming Interface List
Bundle-Ether162.4 Flags: A, Up: 04:20:11
Outgoing Interface List
Bundle-Ether361.6 Flags: F NS, Up: 04:20:11
RP/0/RP0/CPU0:Router#
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 101
NCS 5500 System Architecture
Control Plane
• The process running on RP CPU will dynamically compute two tables for each MCID
• MCID mapping is a 128 bitmap mask where Ones represent NPUs who received a
join and who expect a copy of the packet from the fabric
• MCID-DB associates ports where a
MCID 60414
RP CPU
replication is expected IGMP/PIM
process
LC1
Hu0/1/0/0
NIF
NPU-0
Hu0/1/0/5 Fabric Egress LC
MCID-Mapping MCID-DB
NIF
Hu0/1/0/7 NPU-1
60414 LC1 NPU0 60414 LC1 Int-0
LC2 NPU0 Int-5
LC1 NPU1
Hu0/2/0/3 LC1
LC2 NPU0 Int-7
NIF
NPU-0 NPU1
Hu0/2/0/4
0000010011..000
LC2 Int-3
IGMP/PIM Join NPU0 Int-4
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
NCS 5500 System Architecture
Show Commands
RP/0/RP0/CPU0:ios#show mrib route detail
<SNIP>
(25.1.1.2,232.1.1.4) Ver: 0x6632 RPF nbr: 25.1.1.2 Flags: RPF, FGID: 3177
Up: 2w4d
Incoming Interface List
BVI1 Flags: A, Up: 2w4d
Outgoing Interface List
TenGigE0/3/0/3/0.100 Flags: F NS LI, Up: 2w4d
RP/0/RP0/CPU0:ios#
RP/0/RP0/CPU0:ios#show mfib route 232.1.1.4 location 0/3/CPU0
(25.1.1.2,232.1.1.4), Flags:
Up: 2w4d
Last Used: never
SW Forwarding Counts: 0/0/0
SW Replication Counts: 0/0/0
SW Failure Counts: 0/0/0/0/0
TenGigE0/3/0/1/0.100 Flags: A, Up:2w4d
TenGigE0/3/0/2/0.200 Flags: NS EG, Up:2w4d
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 103
NCS 5500 System Architecture
MCID Bitmap
RP/0/RP0/CPU0:ios#show mrib fgid info 3177
FGID information
----------------
FGID (type) : 3177 (Primary)
Context : IP (0xe0000000, 25.1.1.2, 232.1.1.4/32)
Members[ref] : 0/3/0[1]
LineCard Slot : 3 :: Npu Instance 0
FGID bitmap
0x0000000000040000 0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000
FGID chkpt context valid : TRUE
FGID chkpt context :
table_id 0xe0000000 group 0xe8010104/32 source 0x19010102
FGID chkpt info : 0x23000000
Fgid in batch : NO
Secondary node count : 0
RP/0/RP0/CPU0:ios#
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 104
NCS 5500 System Architecture
MCID Bitmap
sysadmin-vm:0_RP0# show controller fabric fgid information id 10927 detail
LC1
Forwarding FEC Resolution Fabric NPU-0 Hu0/1/0/0
NIF
Interface egress Hu0/1/0/5
NIF
NIF
Hu0/1/0/7
egress
(VRF, S, G) LC2
Lookup RPF check Hu0/2/0/3
NPU-0
NIF
Ingress Pipeline Fabric Cards egress Hu0/2/0/4
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 106
NCS 5500 System Architecture
Data Plane
• Internal Header has been marked with MCID
NIF
Interface egress Hu0/1/0/5
NIF
NIF
Hu0/1/0/7
egress
(VRF, S, G) LC2
Lookup RPF check Hu0/2/0/3
NPU-0
NIF
Ingress Pipeline Fabric Cards egress Hu0/2/0/4
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 107
NCS 5500 System Architecture
Data Plane
MCID-DB
• Re-assembled packets 60414 LC1 NPU0 Int-0
will be replicated on Int-5
Hu0/1/0/0
egress NPU based on NPU-0
LC1
NIF
egress
MCID-DB information Hu0/1/0/5
LC1
ingress NPU-1
NIF
Hu0/1/0/7
egress
MCID-DB
LC2
NPU-0
NIF
egress
Fabric Cards Hu0/2/0/4
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 108
Multicast Packet Queueing in NCS 5500
• Based on Fabric Multicast Queues
• Pairs of Traffic Class mapped into FMQ
• TC 0 and 1 to FMQ 0
• TC 2 and 3 to FMQ 1
• TC 4 and 5 to FMQ 2
• TC 6 and 7 to FMQ 3
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 109
Multicast Packet Queueing in NCS 5500
Fabric Card 0
Fabric Card 3
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 110
Multicast Packet Queueing in NCS 5500
Fabric Card 0
Fabric Card 3
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 111
Multicast Packet Queueing in NCS 5500
Fabric Card 0
Fabric Card 3
policy-map
• Ingress Traffic Manager selects packet from an FMQ and gives it to Ingress Fab
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 112
Multicast Packet Queueing in NCS 5500
Fabric Card 0
Fabric Card 3
• Ingress Fab splits packet into cells and load balances them across the fabric cards
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 113
Multicast Packet Queueing in NCS 5500
Fabric Card 0
Fabric Card 3
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 114
Multicast Packet Queueing in NCS 5500
Fabric Card 0
Egress Egress
Ingress Fab Interface Net Interfaces
Fabric Fabric Card 1
Interface Queues
Multicast
Queues Not
Fabric Card 2
Net Fab
controlled
by output
Fabric Card 3
policy-map
Fabric Card 4
Egress
Fab Interface Net Egress
Queues Interface
Fabric Card 5
Unicast LP
Fabric Card 3
Multicast LP
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 116
NCS 5500 Internals
NCS 5500 System Architecture
Intra-Chassis Communication
• EOBC and EPC: two isolated networks
GMAC0 GMAC0
LC0-7 FC0-5
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 119
RP0 RP1
NCS 5500 Internals GMAC0 GMAC1 GMAC0 GMAC1
EPC in Modular Chassis
• Ethernet Protocol Channel
• Intra-system data plane protocol SC0 EPC EPC
Switch Switch SC1
communication
• EPC switch only connects Fabric
Cards to RPs EPC
• If protocol packets need to be sent Switch
FC0-5
to RP, line cards utilize the internal
data path to transfer packets to
Fabric Cards first, Fabric Cards
then redirect them via the EPC EPC
Switch
channel to supervisor engines
• Uses different VLAN for different GMAC0 LC0-7
traffic types (one VLAN per NPU for
Netflow sampled packets)
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 120
For Reference
sysadmin-vm:0_RP0#
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 121
NCS 5500 Internals
EPC/EOBC Switches FC0-5 SC0 SC1
• In Line Cards, switches EPC SC0 SC1
Switch EOBC EOBC
are shared for EPC/EOBC Switch Switch
• Different bandwidth
depending on the LC type
EPC
(1G, 2.5G)
EOBC+EPC
• Only one Fabric Card link LC EOBC
EPC EOBC
is forwarding Switch Switch CPU down
NPU5
NPU1
NPU2
NPU3
NPU4
NPU0 122
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Example: EPC/EOBC in 24x100G Line Cards
sysadmin-vm:0_RP0# show controller switch summary location 0/LC7/LC-SW FC0-5 SC0 SC1
Rack Card Switch Rack Serial Number
EPC SC0 SC1
--------------------------------------
0 LC7 LC-SW FGE194714QQ Switch EOBC EOBC
Switch Switch
Phys Admin Port Protocol Forward
Port State State Speed State State Connects To
--------------------------------------------------------------------
4 Up Up 2.5-Gbps - Forwarding LC CPU (EPC 0)
5 Up Up 2.5-Gbps - Forwarding LC CPU (EPC 1)
6 Up Up 2.5-Gbps - Forwarding LC CPU (EPC 2)
7 Up Up 2.5-Gbps - Forwarding LC CPU (EOBC)
8 Up Up 2.5-Gbps - Forwarding NPU2
9 Up Up 2.5-Gbps - Forwarding NPU1 EPC EOBC LC
10 Up Up 2.5-Gbps - Forwarding NPU0 Switch Switch CPU
11 Up Up 2.5-Gbps - Forwarding NPU3
12 Up Up 1-Gbps - Forwarding FC0
13 Down Down 1-Gbps - - FC1
NPU1
NPU2
NPU3
NPU0
14 Down Down 1-Gbps - - FC2
15 Down Down 1-Gbps - - FC3
16 Down Down 1-Gbps - - FC4
17 Down Down 1-Gbps - - FC5
18 Up Up 1-Gbps - Forwarding SC0 EOBC-SW
19 Down Down 1-Gbps - - SC1 EOBC-SW
sysadmin-vm:0_RP0#
EOBC EPC
EOBC+EPC
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
NCS 5500
Memory Structure
Route Scale per Platform
Hardware Scale
NCS5501 1.1M pfx
NCS5501-SE 2.75M pfx
NCS5502 1.1M pfx
NCS5502-SE 2.75M pfx
NCS55A1-36H-S 1.1M pfx
NCS55A1-36H-SE-S 4M pfx
NCS55A1-24H 2M+ pfx
NCS55A2-MOD-S 1.1M pfx
NCS55A2-MOD-HD-S 1.1M pfx
NCS55A2-MOD-SE-S 4M pfx
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 125
Route Scale per Platform
Hardware Scale
NC55-36X100G 1.1M pfx
NC55-24X100G-SE 2.75M pfx
NC55-18H18F 1.1M pfx
NC55-24H12F-SE 2.75M pfx
NC55-36X100G-S 1.1M pfx
NC55-6x200-DWDM-S 1.1M pfx
NC55-36X100G-A-SE 4M pfx
NC55-MOD-A-S 1.1M pfx
NC55-MOD-A-SE-S 4M pfx
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 126
NCS 5500 Forwarding ASIC Details
Memory / Databases
• Longest Prefix Match Database (LPM or KAPS)
• Used to store IPv4 and IPv6 prefixes
• Algorithmic memory: worst case 256k entries (IPv6 uses 2 entries)
• Large Exact Match Database (LEM)
• Used to store MAC addresses, MPLS labels and IPv4 host prefix
(but also /24, /23, /20… Database size: 786k entries)
• Internal TCAM (iTCAM)
• Packet classification (ACL, QoS, VLAN ranges, tunnels. Database size: 48k entries)
• External TCAM (eTCAM, not on all line cards / systems)
• Used for unicast route scale up to 2M or 4M IPv4 Routes
• Used to extend ACL and classification
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 127
NCS 5500 Forwarding ASIC Details
Algorithmic Database (LPM) ?
• LPM memory is qualified for 256k IPv4 or 128k IPv6 addresses worst case
Table Configuration
<SNIP>
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 128
NCS 5500 Forwarding ASIC Details
Algorithmic Database – Specific Case of NCS55A1-24H
• The 1RU 24x100G system uses a specific flavor of Forwarding ASIC
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 129
NCS 5500 Forwarding ASIC Details
Memory / Databases
• FEC
• Used for NextHop and ECMP (128k entries)
• Contains the FEC ECMP (4k entries)
• Egress Encapsulation DB (EEDB)
• Used for egress rewrites (96k entries)
• Link Local – ARP, ND
• Tunnel – MPLS label, GRE, etc
• FEC Entry contains VOQ / Egress Interface and EEDB (encapsulation entry)
• EEDB indicates the encapsulation for the packet (ARP, ND or GRE, MPLS, …)
Forwarding FEC Resolution Header Editor Encap Editor
LEM
FEC EEDB
FABRIC
LPM
ECMP
FEC
eTCAM
Next-Hop
Prefixes Load-balancing
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 132
Memory Structure for Non-eTCAM Systems / LC
Public Internet View: 58966 v6 routes
• Hiding IPv6 with 0 entry
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 133
Memory Structure for Non-eTCAM Systems / LC
Host Optimized Mode
MPLS LEM
MAC Lookup
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 134
Non-eTCAM Systems / LC Host Optimized Mode
Illustration with Public Internet View: 655815 v4 and 58966 v6 real routes
HW Resource Information
Name : lem
v4/32 and v4/24
Current Usage
NPU-0
Total In-Use : 386610 (49 %) v6/48
iproute : 367385 (47 %)
ip6route : 19222 (2 %)
mplslabel : 5 (0 %)
HW Resource Information
Name : lpm
Other v4 routes
Current Usage
NPU-0
Total In-Use : 328236 (83 %) Other v6 routes
iproute : 288456 (73 %)
ip6route : 39767 (10 %)
ipmcroute : 0 (0 %)
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 135
Memory Structure for Jericho non-eTCAM
Internet Optimized Mode
LPM LEM LEM LPM
LPM IPv4 prefixes IPv4 Lookup 1 Lookup 1 Lookup 2 Lookup 2
256k-350K
(except those in LEM)
IPv6 prefixes (non-/48s) /32 /25 /24 and /23 /20 /22, /21, /20
entries
Multicast groups v4 /19 /0
z
LEM IPv4 prefixes (/20s, /23s - /24s)
IPv6 prefixes (/48s)
786k
entries
MPLS labels
MAC addresses LPM LEM LPM
IPv6 Lookup 1 Lookup Lookup 2
MPLS LEM
MAC Lookup
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 136
Non-eTCAM Systems with Internet Optimized Mode
Illustration with Public Internet View: 655815 v4 and 58966 v6 real routes
HW Resource Information
Name : lem
v4/24, v4/23 expanded
Current Usage v4/20
NPU-0
Total In-Use : 530670 (67 %) v6/48
iproute : 518495 (66 %)
ip6route : 19222 (2 %)
mplslabel : 5 (0 %)
HW Resource Information
Name : lpm
Other v4 routes
Current Usage v4/20 with overlaps
NPU-0
Total In-Use : 231172 (51 %) Other v6 routes
iproute : 194021 (43 %)
ip6route : 39768 (9 %)
ipmcroute : 0 (0 %)
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 137
Memory Structure for eTCAM Systems / LC
Default Distribution
eTCAM
2M IPv4 pfx (non /32s) MPLS LEM
entries MAC Lookup
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 138
Memory Structure for J w/ eTCAM
Illustration with Public Internet View: 655815 v4 and 58966 v6 real routes
HW Resource Information
Name : lem
v4/32
Current Usage
NPU-0 v6/48
Total In-Use : 20132 (3 %)
iproute : 904 (0 %)
ip6route : 19222 (2 %)
mplslabel : 5 (0 %)
HW Resource Information
Name : lpm
No v4 routes in LPM
Current Usage
NPU-0
Total In-Use : 39786 (10 %) Other v6 routes
iproute : 0 (0 %)
ip6route : 39767 (10 %)
ipmcroute : 0 (0 %)
HW Resource Information
Name : ext_tcam_ipv4
All v4 routes
Current Usage
NPU-0 except v4/32
Total In-Use : 654937 (40 %)
iproute : 654937 (40 %)
ipmcroute : 0 (0 %)
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 139
Memory Structure for J+ w/ eTCAM Systems / LC
Starting from IOS XR 6.3.2
LPM
eTCAM
4M IPv4 + IPv6 pfx
entries
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 140
For Reference
Demos
http://bit.ly/ncs5500-base http://bit.ly/ncs5500-scale
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 141
NCS 5500
TCAM Carving
Default eTCAM Carving
Jericho w/ eTCAM
RP/0/RP0/CPU0:NCS5508-6.3.2#sh contr npu ext loc 0/6/CPU0 RP/0/RP0/CPU0:TME-5508-6.2.3#sh contr npu externaltcam loc 0/6/CPU0
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 143
Default eTCAM Carving
Jericho w/ eTCAM with URPF Loose
• Activating URPF requires to disable the eTCAM dual capacity mode
80b 80b
IPv4 Route IPv4 Route IPv4 Route IPv4 Route
IPv4 Route IPv4 Route IPv4 Route IPv4 Route
IPv4 Route IPv4 Route IPv4 Route IPv4 Route
IPv4 Route IPv4 Route IPv4 Route IPv4 Route
IPv4 Route IPv4 Route IPv4 Route IPv4 Route
IPv4 Route IPv4 Route IPv4 Route IPv4 Route
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 144
Default eTCAM Carving
Jericho w/ eTCAM with URPF Loose
• It effectively reduces the eTCAM size by half
disabled disabled
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 145
Default eTCAM Carving
Jericho+ w/ eTCAM
• In 6.3.2, the system is validated for 4M v4 routes (with or without uRPF)
• Hybrid ACL objects are stored in a different zone and don’t impact the
scale
eTCAM
4M IPv4 / IPv6 pfx
entries
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 146
Modifying eTCAM Carving
Jericho w/ eTCAM
• It’s advised to configure a total of 100% for predictable results
Table Configuration
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 147
Modifying eTCAM Carving
Jericho w/ eTCAM
LEM eTCAM
IPv4 Lookup Lookup
LPM IPv4 Multicast Groups
/32 /31 /0
z
LEM
IPv4 /32s
eTCAM
786k MPLS labels IPv6 Lookup
entries MAC addresses
/128 /0
Only v4/32s are programmed in LEM Configuring 100% IPv6 in eTCAM is not possible,
All other v4/v6 routes go to eTCAM except but 1% / 99% is accepted
if x=100 / y=0, IPv6 will be moved to LEM/LPM
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 148
Monitoring Memory Resources
Thresholds Yellow / Red
• For both base and scale systems
• Each database is using two thresholds: yellow at 80% and red at 95%
LC/0/0/CPU0:Jan 18 23:41:56.750 : fia_driver[279]: %PLATFORM-DPA-1-OOR_RED : NPU 0, Table iproute
LC/0/0/CPU0:Jan 18 23:41:56.750 : fia_driver[279]: %PLATFORM-DPA-4-OOR_YELLOW : NPU 0, Table iproute
LC/0/0/CPU0:Jan 18 23:41:56.750 : fia_driver[279]: %PLATFORM-DPA-1-OOR_RED : NPU 0, Table iproute
LC/0/0/CPU0:Jan 18 23:42:00.336 : fia_driver[279]: %PLATFORM-DPA-1-OOR_RED : NPU 2, Table iproute
LC/0/0/CPU0:Jan 18 23:42:00.418 : fia_driver[279]: %PLATFORM-DPA-1-OOR_RED : NPU 4, Table iproute
LC/0/0/CPU0:Jan 18 23:42:00.438 : fia_driver[279]: %PLATFORM-DPA-4-OOR_YELLOW : NPU 4, Table iproute
LC/0/0/CPU0:Jan 18 23:42:00.439 : fia_driver[279]: %PLATFORM-DPA-1-OOR_RED : NPU 4, Table iproute
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 149
Monitoring Memory Resources
Exceeding a Database Capacity
• DPA will not program new prefixes and “Hw failures” counter will increment
<SNIP>
NPU ID: NPU-0 NPU-1 NPU-2 NPU-3 NPU-4 NPU-5
<SNIP>
Errors
HW Failures: 16131 16131 16131 16132 16131 16131
Resolve Failures: 0 0 0 0 0 0
No memory in DB: 0 0 0 0 0 0
Not found in DB: 0 0 0 0 0 0
Exists in DB: 0 0 0 0 0 0
RP/0/RP0/CPU0:NCS5508#
RP/0/RP0/CPU0:NCS5508#sh contr npu resources lem location 0/0/CPU0
<SNIP>
Current Usage
NPU-0
Total In-Use : 783898 (100 %)
iproute : 783898 (100 %) (Prefix Count: 783898)
mplslabel : 0 (0 %) (Prefix Count: 0)
<SNIP>
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 150
Monitoring Memory Resources
CLI to Check LEM Database Usage
RP/0/RP0/CPU0:5508-6.3.2#sh contr npu resources all loc 0/1/CPU0 <...>
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 151
For Reference
<...>
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 152
For Reference
<...>
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 153
For Reference
<...>
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 154
For Reference
<...>
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 155
For Reference
<...>
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 156
For Reference
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 157
For Reference
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 158
For Reference
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 159
For Reference
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 162
Mixing Scale and Base
LineCards
Selective Route Download Feature
• eTCAM and non-eTCAM can co-exist in the same chassis
• It’s possible to select routes that will be programmed in scale line cards only
• In BGP configuration
LPM LPM
• using a table-policy and a specific path-color “external-reach”
256k-350K 256k-350K
• With this feature entries entries
eTCAM
2M
entries
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 164
Selective Route Download Configuration
route-policy PEER-EXT
set community PEER-EXT-comm
end-policy
!
route-policy HILO-FIB
if community matches-any PEER-EXT-comm then
set path-color external-reach
pass
else
pass
endif
end-policy!
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 165
Selective Route Download Verification
• Check a route
RP/0/RP0/CPU0:NCS5508-1-631#sh route 1.0.144.0/20
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 166
Selective Route Download Verification
RP/0/RP0/CPU0:NCS5508-1-631#sh cef 1.0.144.0/20 detail
1.0.144.0/20, version 25081094, external-reach-lc-only, internal 0x5000001 0x0 (ptr 0x8f485390) [1], 0x0
(0x0), 0x0 (0x0)
Updated Nov 27 22:48:56.929
local adjacency 192.168.100.151
Prefix Len 20, traffic index 0, precedence n/a, priority 4
gateway array (0x8e0e9250) reference count 655801, flags 0x2010, source rib (7), 0 backups
[1 type 3 flags 0x48501 (0x8e18f758) ext 0x0 (0x0)]
LW-LDI[type=0, refc=0, ptr=0x0, sh-ldi=0x0]
gateway array update type-time 1 Nov 27 22:48:56.929
LDI Update time Nov 27 22:48:56.929
via 192.168.100.151/32, 2 dependencies, recursive [flags 0x6000]
path-idx 0 NHID 0x0 [0x8e0bf1b0 0x0]
next hop 192.168.100.151/32 via 192.168.100.151/32
RP/0/RP0/CPU0:NCS5508-1-631#
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 167
Selective Route Download Use-Case
• Lookup executed in ingress only
• Position of the Base and Scale line card is opposite than ASR9k or CRS
• Internet-facing interface could be DWDM card or MACsec card
Content
Servers DC role Peering role
Scale Base MPLS Scale Base
LC LC Core LC LC Internet
Internal MPLS
All Only
+ all and
Internet Internal
Internet Customer
Routes Routes
Routes Routes
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 168
For Reference
Demo
http://bit.ly/ncs5500-mix
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 169
NCS 5500
Access-Lists
Using Access-Lists
With Jericho and Jericho+ LC / Systems
• Traditional ACLs
• Supported on systems with or without eTCAM
• ACEs are stored in iTCAM only
• Hybrid / Scale ACLs
• Supported on scale systems only (with eTCAM)
• Part of the ACE will be stored and compress on eTCAM
• Other part of the ACE will be in iTCAM (2-step look-up mechanism)
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 171
Traditional ACLs
Using Only Internal TCAM (iTCAM)
• 12 large banks (0-11): 2k entries each
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 172
Traditional ACLs
Checking Internal TCAM (iTCAM) in 6.2.2 Onwards
RP/0/RP0/CPU0:NCS5508-2-622#sh contr npu internaltcam location 0/7/CPU0
Internal TCAM Resource Information
NPU Bank Entry Owner Free Per-DB DB DB
Id Size Entries Entry ID Name
=============================================================
0 0\1 320b pmf-0 2006 36 7 INGRESS_LPTS_IPV4
0 0\1 320b pmf-0 2006 2 12 INGRESS_RX_ISIS
0 0\1 320b pmf-0 2006 2 32 INGRESS_QOS_IPV6
0 0\1 320b pmf-0 2006 2 34 INGRESS_QOS_L2
0 2 160b pmf-0 2044 2 31 INGRESS_QOS_IPV4
0 2 160b pmf-0 2044 1 33 INGRESS_QOS_MPLS
0 2 160b pmf-0 2044 1 42 INGRESS_ACL_L2
0 3 160b egress_acl 2022 10 3 EGRESS_RECEIVE
0 3 160b egress_acl 2022 16 4 EGRESS_QOS_MAP
0 4\5 320b pmf-0 2024 24 8 INGRESS_LPTS_IPV6 Free Space
0 6 160b Free 2048 0 0 No ACL configured
0 7 160b Free 2048 0 0
0 8 160b Free 2048 0 0
0 9 160b Free 2048 0 0
0 10 160b Free 2048 0 0
0 11 160b Free 2048 0 0
0 12 160b pmf-1 90 37 11 INGRESS_RX_L2
0 12 160b pmf-1 90 1 13 INGRESS_MCAST_IPV4_ASM
0 13 160b pmf-0 112 2 10 INGRESS_DHCP
0 13 160b pmf-0 112 13 26 INGRESS_MPLS
0 13 160b pmf-0 112 1 41 INGRESS_EVPN_AA_ESI_TO_FBN_DB
0 14 160b Free 128 0 0
0 15 160b Free 128 0 0
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 173
Traditional ACLs
Checking Internal TCAM (iTCAM) in 6.2.2 Onwards
RP/0/RP0/CPU0:NCS5508-2-622#sh contr npu internaltcam location 0/7/CPU0
Internal TCAM Resource Information
NPU Bank Entry Owner Free Per-DB DB DB
Id Size Entries Entry ID Name
=============================================================
0 0\1 320b pmf-0 2006 36 7 INGRESS_LPTS_IPV4
0 0\1 320b pmf-0 2006 2 12 INGRESS_RX_ISIS
0 0\1 320b pmf-0 2006 2 32 INGRESS_QOS_IPV6
0 0\1 320b pmf-0 2006 2 34 INGRESS_QOS_L2
0 2 160b pmf-0 2044 2 31 INGRESS_QOS_IPV4
0 2 160b pmf-0 2044 1 33 INGRESS_QOS_MPLS
0 2 160b pmf-0 2044 1 42 INGRESS_ACL_L2
0 3 160b egress_acl 2022 10 3 EGRESS_RECEIVE
0
0
3
4\5
160b
320b
egress_acl 2022
pmf-0 2024
16
24
4
8
EGRESS_QOS_MAP
INGRESS_LPTS_IPV6
1000 ACEs configured
0 6 160b pmf-0 997 1051 16 INGRESS_ACL_L3_IPV4
0 7 160b Free 2048 0 0
0 8 160b Free 2048 0 0
0 9 160b Free 2048 0 0
0 10 160b Free 2048 0 0
0 11 160b Free 2048 0 0
0 12 160b pmf-1 90 37 11 INGRESS_RX_L2
0 12 160b pmf-1 90 1 13 INGRESS_MCAST_IPV4_ASM
0 13 160b pmf-0 112 2 10 INGRESS_DHCP
0 13 160b pmf-0 112 13 26 INGRESS_MPLS
0 13 160b pmf-0 112 1 41 INGRESS_EVPN_AA_ESI_TO_FBN_DB
0 14 160b Free 128 0 0
0 15 160b Free 128 0 0
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 174
Traditional ACLs
Counters
• Limitations with packets targeted to the router
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 175
Traditional ACLs
Counting with permit ACEs
• By default only deny ACEs are allocated counters
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 176
object-group network ipv4 netobj1
description my-network-object
v4 Pfx 80%
iTCAM
ACL 20%
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 177
Hybrid ACLs
Example
• CLI to display an RP/0/RP0/CPU0:R1#sh access-lists ipv4 network-object-acl
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 178
Hybrid ACLs
Monitoring Resource: 1- On eTCAM
RP/0/RP0/CPU0:NCS5508-1-631#sh contr npu externaltcam loc 0/7/CPU0
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 179
Hybrid ACLs
Monitoring Resource: 2- On iTCAM
RP/0/RP0/CPU0:NCS5508-1-631#sh contr npu internaltcam loc 0/7/CPU0
Internal TCAM Resource Information
=============================================================
NPU Bank Entry Owner Free Per-DB DB DB
Id Size Entries Entry ID Name
=============================================================
0 0\1 320b pmf-0 1963 49 7 INGRESS_LPTS_IPV4
0 0\1 320b pmf-0 1963 2 12 INGRESS_RX_ISIS
0 0\1 320b pmf-0 1963 11 32 INGRESS_QOS_IPV6
0 0\1 320b pmf-0 1963 23 34 INGRESS_QOS_L2
0 2 160b pmf-0 2030 11 31 INGRESS_QOS_IPV4
0 2 160b pmf-0 2030 6 33 INGRESS_QOS_MPLS
0 2 160b pmf-0 2030 1 42 INGRESS_ACL_L2
0 3 160b egress_acl 2032 16 4 EGRESS_QOS_MAP
0 4\5 320b pmf-0 2021 27 8 INGRESS_LPTS_IPV6 `
0 6\7 320b pmf-1 2045 3 49 INGRESS_HYBRID_ACL
0 8 160b Free 2048 0 0
0 9 160b Free 2048 0 0
0 10 160b Free 2048 0 0
0 11 160b Free 2048 0 0
0 12 160b pmf-1 88 40 11 INGRESS_RX_L2
0 13 160b pmf-0 84 3 10 INGRESS_DHCP
0 13 160b pmf-0 84 1 13 INGRESS_MCAST_IPV4_ASM
0 13 160b pmf-0 84 13 26 INGRESS_MPLS
0 13 160b pmf-0 84 1 41 INGRESS_EVPN_AA_ESI_TO_FBN_DB
0 13 160b pmf-0 84 26 79 INGRESS_BFD_IPV4_NO_DESC_TCAM_T
0 14 160b Free 128 0 0
0 15 160b Free 128 0 0
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 180
NCS 5500 QoS
Quality of Service on NCS5500
• Ingress direction supports classification and remarking
• Ingress direction supports only policing
• Egress direction supports only shaping
Ingress Egress
Policing Queueing
config config
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 182
Quality of Service
Internal Markers
• We use internal markers at ingress to take egress actions
Ingress Egress
match xxx
match qos-group Egress Remarking
set qos-group
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 183
Configuring Quality of Service
Policer Configuration
Ingress Egress
class-map classify1
match precedence 1 Class-Map
policy-map Pol1
Match criterias
class classify1
set qos-group 1
set dscp ef
police rate percent 10 set qos-group (optional)
interface hu 0/0/0/0
(optional)
service-policy input Pol1
set dscp/…
Policer
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 184
For Reference
class-map classify1
match precedence 1
class-map classify2
match precedence 2
class-map classify3
30Gbps Prec 1
match precedence 3 qos-group 1
20Gbps Prec 1
policy-map ingress-policy
class classify1 10Gbps Prec 2 qos-group 2 10Gbps Prec 2
set qos-group 1
qos-group 3
police rate percent 10 peak-rate percent 20 10Gbps Prec 3 10Gbps Prec 3
class classify2 qos-group 0
set qos-group 2 10Gbps Prec 4 10Gbps Prec 4
class classify3
set qos-group 3
interface hu 0/0/0/0
service-policy input ingress-policy
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 185
Configuring Quality of Service
Shaper Configuration
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 187
Configuring Quality of Service
Shaper Configuration on Bundles
• All QoS rules applied to a bundle are applied to all members
BE100 BE100
Priority1: 10%
Hu0/1/0/0
Queue3:25% Queue2: 50% goes down Queue3:25%
Hu0/1/0/0
Queue3:25%
Default: 15% Default: 15%
Default: 15%
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 188
Configuring Quality of Service
Shaper Configuration on Bundles
• If we use absolute values, they are applied to each member too
Use percent
Priority1: 5G Priority1: 5G
Priority1: 5G Priority1: 5G
BE100 BE100
Priority1: 5G
Hu0/1/0/0
Queue3: 12G Queue2: 25G goes down Queue3: 12G
Hu0/1/0/0
Queue3: 12G
Default: 7G Default: 7G
Default: 7G
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 189
Key Differences with Traditional XR Platforms
Unicast is Scheduled but Multicast Traffic doesn’t Follow VOQ-only Model
• In case of egress interface congestion
• If unicast or multicast is high priority, it will take full precedence over the other
• If same priority (HP/HP or LP/LP), then the forwarding will be 80% ucast / 20% mcast
10G10G 10G 10G 10G 10G
HP LP LP LP LP HP
10G 10G
2G
8G BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
NCS 5500 Software
Architecture
Software Architecture
Classic and Containerized OS
• IOS XR Exists in two flavours
in NCS 6000
• Linux based IOS XR Admin
IOS XR
• Larger addressable memory LxC LxC
[xr-vm_node0_RP0_CPU0:~]$
[xr-vm_node0_RP0_CPU0:~]$exit XR Guest OS
exit Admin
RP/0/RP0/CPU0:LAB#admin IOS XR
LxC
Mon Nov 16 21:27:05.958 UTC LxC
(calvados)
[sysadmin-vm:0_RP0:~]$
[sysadmin-vm:0_RP0:~]$ssh 10.0.2.16 admin Guest OS
[host:0_RP0:~]$ admin Host OS
[host:0_RP0:~]$
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 193
For Reference
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 194
OS Structure on Modular Platforms
NCS 5508: Based on LXCs and not VMs
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 195
For Reference
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 196
OS Structure on Fixed Platforms
NCS 5501 and NCS 5502
• NCS 5501 is a SoC architecture, everything operates in one CPU
• NCS 5502 is actually a distributed platform, we will use a container for the LC
NCS 5501 / NCS 5502
admin LineCard
IOS XR 3rd Party
LXC IOS XR
LXC LXC
(calvados) LXC
Host OS
Wind River 7, linux kernel 3.14
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 197
Conclusion
Conclusion
• Merchant silicon is not something new in SP portfolio
• Port density and features
• NCS 5500 can be used in multiple roles in Networks such as
• Core, Peering, SP DC, Aggregation and Edge: You decide.
• Architecture based on VOQ-only for unicast and FMQ for multicast
• Compared to traditional IOS XR platforms
• Resources needs to be monitored differently
• Features can have a different implementation
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 199
Cisco Webex Teams
Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session
How
1 Find this session in the Cisco Events Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 200
Complete your online
session survey
• Please complete your Online Session
Survey after each session
• Complete 4 Session Surveys & the Overall
Conference Survey (available from
Thursday) to receive your Cisco Live T-
shirt
• All surveys can be completed via the Cisco
Events Mobile App or the Communication
Stations
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 201
Continue Your Education
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 202
Thank you
BackUp Slides
CEF
NCS 5500 CEF Implementation
Case 1: Non-Recursive / No-ECMP
RP/0/RP0/CPU0:R1#sh cef 4.4.4.4 hardware egress location 0/0/CPU0
4.4.4.4/32, version 304, internal 0x1000001 0x0 (ptr 0x8908a698) [3], 0x0 (0x88f74868), 0x0 (0x0)
Updated Jan 11 14:31:40.966
local adjacency 192.1.2.2
Prefix Len 32, traffic index 0, precedence n/a, priority 1
FEC Handle
via 192.1.2.2/32, HundredGigE0/0/0/46, 5 dependencies, weight 0, class 0 [flags 0x0]
path-idx 0 NHID 0x0 [0x89d760a0 0x0]
next hop 192.1.2.2/32 Since it is a non-recursive
local adjacency
lookup, the FEC location is in
LEAF - HAL pd context :
sub-type : IPV4, ecd_marked:0, has_collapsed_ldi:0
SHLDI.
collapse_bwalk_required:0, ecdv2_marked:0
HW Walk:
LEAF: FEC Index and the number of
PI:0x308908a698 PD:0x308908a730 Rev:1198 type: 0
FEC handle: 0x308936b248 paths available as next hop.
LWLDI:
PI:0x3088f74868 PD:0x3088f748a8 rev:1197 p-rev:1187 ldi type:3
FEC hdl: 0x308936b248 fec index: 0x0(0) num paths:1, bkup: 0
FEC Index points to DSP
which is the pointer to the
SHLDI:
PI:0x3088df2068 PD:0x3088df20e8 rev:1187 dpa-rev:6925889 flag:0x0 next hop. In this example,
FEC hdl: 0x308936b248 fec index: 0x2000100a(4106) num paths: 1 bkup paths: 0
p-rev:1129
the VOQ 0x570 = 1392.
Path:0 fec index: 0x2000100a(4106) DSP:0x570 Dest fec index: 0x0(0)
RP/0/RP0/CPU0:R1# © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
NCS 5500 CEF Implementation
Case 1: Non-Recursive / No-ECMP
RP/0/RP0/CPU0:R1#sh contr npu voq-usage interface all instance all location 0/0/CPU0 | i 1392
Hu0/0/0/46 300 7 0 5 453 1392 1336 local 100 Local: location 0/0/cpu0
RP/0/RP0/CPU0:R1#
Port 46: NPU number 7
RP/0/RP0/CPU0:R1#show controllers fia diagshell 7 "diag pp LIF_show type=out id=0x40010002 gl=1"
location 0/0/CPU0
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
NCS 5500 CEF Implementation
Case 1: Non-Recursive / No-ECMP
RP/0/RP0/CPU0:Router#sh cef 4.4.4.4 hardware egress location 0/0/CPU0
4.4.4.4/32, version 304, internal 0x1000001 0x0 (ptr 0x8908a698) [3], 0x0 (0x88f74868), 0x0 (0x0)
Updated Jan 11 14:31:40.966
local adjacency 192.1.2.2
Prefix Len 32, traffic index 0, precedence n/a, priority 1
via 192.1.2.2/32, HundredGigE0/0/0/46, 5 dependencies, weight 0, class 0 [flags 0x0]
path-idx 0 NHID 0x0 [0x89d760a0 0x0]
next hop 192.1.2.2/32
local adjacency
Hardware Walk
Let’s focus on this part
of the show cef CLI
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 208
NCS 5500 CEF Implementation
Case 1: Non-Recursive / No-ECMP
FEC Handle Index:0 Paths: 1 FEC Index
Non-recursive SHLDI No ECMP
HW Walk:
LEAF:
PI:0x308908a698 PD:0x308908a730 Rev:1198 type: 0
FEC handle: 0x308936b248
DSP
Pointer
LWLDI:
PI:0x3088f74868 PD:0x3088f748a8 rev:1197 p-rev:1187 ldi type:3 to NH
FEC hdl: 0x308936b248 fec index: 0x0(0) num paths:1, bkup: 0
SHLDI: Encap id
PI:0x3088df2068 PD:0x3088df20e8 rev:1187 dpa-rev:6925889 flag:0x0
FEC hdl: 0x308936b248 fec index: 0x2000100a(4106) num paths: 1 bkup paths: 0 In EEDB
p-rev:1129
Path:0 fec index: 0x2000100a(4106) DSP:0x570 Dest fec index: 0x0(0)
Egress intf
TX-NHINFO:
PD: 0x3089d76118 rev: 1129 dpa-rev: 6919550 Encap hdl: 0x3089e3b0e8
Encap id: 0x40010002 Remote: 0 L3 int: 1048 npu_mask: 80
RP/0/RP0/CPU0:Router#
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 209
NCS 5500 CEF Implementation
Case 1: Non-Recursive / No-ECMP
Encap id ASIC Instance
In EEDB Identified with egress
RP/0/RP0/CPU0:R1#show controllers fia diagshell 7 intf pp LIF_show type=out
"diag
id=0x40010002 gl=1" location 0/0/CPU0 Local Out-LIF
Node ID: 0/0/CPU0
************************************************
Warning: Core 0: Packet diagnostics refers to OLP packet
************************************************
Global Out_LIF:0x40010002 -> Local Out_LIF:0x00004008 -> Type:(null) Bank:2 Offset:4 Destination
LL Encapsulation: MAC
^Mdest_mac:0c:11:67:46:ac:30
^Mout_vid_valid: 1
Address
Linked to this
^Mout_vid: 1048 encap ID
^Mpcp_dei_valid: 0
^Mpcp_dei: 0
^Mtpid_index: 0
^Mll_remark_profile: 0
^Mout_ac_valid: 0
^Mout_ac_lsb: 0
^Moam_lif_set: 0
^Moutlif_profile: 0x10
^MNext_eep: 0x0
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 210
NCS 5500 CEF Implementation
Case 1: Non-Recursive / No-ECMP
IRPP ETPP
SHLDI (non-recursive)
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 211
BackUp Slides
URPF
Generic Comments on uRPF
• Only uRPF Loose mode supported (not Strict mode)
• Loose mode is still useful in 2017 or later?
• How many packets are actually dropped in loose mode?
• Source-Based RTBH is a valid use-case
• Enabled at interface level
• Deactivating uRPF
• Involves disabling it for both IPv4 and IPv6 on the interface
• allow-default is not supported
• allow-self-ping is default mode
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 213
MPLS
Using uRPF Jericho w/o eTCAM MAC
LEM
Lookup
BRKSPG-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 215