IOS-XE Troubleshooting

Hands-on Lab
Olivier Pelerin, Technical Leader
Michal Stanczyk, Customer Support Engineer
Wen Zhang, Technical Leader

LTRARC-3500
Agenda
Introduction to IOS-XE Platform Software/Hardware Architecture
Day in the Life of a Normal Packet
Troubleshooting strategy and Tools
Resource Consumption Monitoring
Understanding and Extracting Platform Logs
Embedded Packet Capture
Data Plane Packet Tracing
Advanced Data Pane Debugging

Hands-on Lab exercise

Wrapping up...
Session Objectives
To understand the IOS-XE (ASR1k, ISR4k, CSR1Kv) platform architecture
Software
Hardware
Feature implementations
Understand how features process packets through IOS-XE
To demonstrate a systematic troubleshooting strategy
To showcase various troubleshooting Tools and Capabilities
To provide a hands-on experience on how to effectively troubleshoot the
platform using these tools

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Related Sessions
BRKCRS-3147 - Advanced troubleshooting of the ASR1K and ISR (IOS-XE)
made easy
Olivier Pelerin Technical Leader, Services
Frederic Detienne Distinguished Engineer, Services

LABRST-2400 - Packet Capturing Tools in Routing Environments WISP Lab

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
ASR Series Hardware Architecture
ASR1K Building Blocks
ESP RP RP ESP
CPU CPU

Active
FECP FECP

Stby
Active

Stby
interconn. GE switch interconn. GE switch QFP
Crypto Crypto
QFP
Assist. PPE BQS Assist. PPE BQS
Route Processor
interconn. Handles control plane traffic interconn.
Manages system
Embedded Service Processor
Handles forwarding plane traffic Midplane

SIP interconn. SIP interconn. SIP interconn.

SPA SPA SPA

IOCP IOCP IOCP
Aggreg. Aggreg. Aggreg.

SPA SPA SPA SPA SPA SPA

SPA Interface Processor

Houses SPAs
Queues packets in & out (FIFO)

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
 System Architecture Control Plane
ESP RP RP ESP
CPU CPU

Active
FECP FECP

Stby
Active

Stby
QFP interconn. GE switch interconn. GE switch QFP
Crypto Crypto
Assist. PPE BQS Assist. PPE BQS
EOBC switch in RP

interconn. interconn.

Midplane
Ethernet Out of Band Channel
(aka EOBC)
Inter Integrated Circuit (I2C) Bus 1Gbps Ethernet bus
SIP interconn. SIP interconn. SIP interconn. Used by RP to program system
Slow (few kbps)
Used for system monitoring Used by system to notify RP
(temp., OIR, fan speed,)
SPA SPA SPA
IOCP IOCP IOCP
Aggreg. Aggreg. Aggreg.

SPA SPA SPA SPA SPA SPA

SPA Control Link

Works between the SPAs and SIP

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
System Architecture Forwarding Plane
Hypertransport
10 Gbps Ethernet
ESP RP RP ESP
CPU CPU

Active
FECP FECP

Stby
Active

Stby
QFP interconn. GE switch interconn. GE switch QFP
Crypto Crypto
Assist. PPE BQS Assist. PPE BQS
Embedded Service Interconnect
aka ESI Bus
interconn. 11.2 200 Gbps Forwarding Bus interconn.

Centralized Architecture Midplane

All traffic flows through ESP

SIP interconn. SIP interconn. SIP interconn.

SPA SPA SPA

IOCP IOCP IOCP
Aggreg. Aggreg. Aggreg.

SPA SPA SPA SPA SPA SPA

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
 RP

Route Processor Architecture

CPU

interconn. GE switch
Highly Scalable Control Plane Processor
Route Processor System
Manages all chassis functions Not a traffic interface! Logging
Runs IOS Management only Core Dumps

Mgmt Console BITS

USB 2.5
Ethernet & Aux (input & output)
Hard disk
Card Infrastructure
Runs IOS, Linux OS
Manages boards and chassis
33MB
IOS Memory: RIB, FIB & NVRAM
other processes RP1: 1GB
Determines BGP routing CPU CPU Bootdisk RP2: 2GB
table size
RP1: 4GB Memory (1.5 2.66 GHz Dual-core)
RP2: 8&16GB Stratum-3 Network
clock circuit GE, 1Gbps
I2C Chassis I2C
Management Bus ESI EOBC SPA Control
Interconnect Gig Eth Switch Output Input SPA Bus

clocks clocks
ESI, 11.2-40 Gbps
SPA-SPI, 11.2Gbps
Hypertransport, 10Gbps
Other
SIPs ESPs RP Misc ESPs SIPs ESPs RP SIPs SIPs RP
Ctrl

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
 ESP
FECP

ESP-xx Block Diagram

QFP
Crypto
Assist. PPE BQS

intercon.

Reset / Pwr Ctrl Packet Buffer

TCAM Resource DRAM Part Len / BW
DRAM
(10Mbit) (512MB) SRAM
Temp Sensor (128MB)

EEPROM
QFP
DDRAM Packet Processor Engine BQS

Boot Flash
(OBFL,) FECP E-CSR
PPE1 PPE2 PPE3 PPE4 PPE5

JTAG Ctrl PPE6 PPE7 PPE8 PPE40

PCI* E-RP*
Dispatcher
GE, 1Gbps
Packet Buffer
I 2C
SPA Control
Crypto SPA Bus
(Nitrox-II CN2430) SPI Mux
Reset / Pwr Ctrl ESI, 11.2Gbps
SA table SPA-SPI, 11.2Gbps
DRAM Interconnect Interconnect Hypertransport, 10Gbps
Other

RPs RPs ESP RPs SIPs

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
ISR Series Hardware Architecture
ISR 4451-X Hardware Diagram

DDR3 Control Plane 4xPCIe Data Plane 4xSGMI

DRAM (4 cores) (10 core) FPGE

Ctrl SVC1 PPE1 PPE2 PPE3 PPE4 PPE5

DDR3
DRAM
SVC2 SVC3 PPE6 PPE7 PPE8 PPE9 PPE10

10 Gbps XAUI

System 1xSGMI
FPGA DSP
Mgmt Ethernet Multi Gigabit
Fabric
Console / Aux 10 Gbps/slot
Peripheral SM-X
Interconnect SM-X
USB
2Gb/slot
Flash

NIM
NIM
NIM

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
ISR 4451-X Hardware Diagram
10 Cores, 1 thread / core
5 fwd cores by default
4 remaining cores license
activated
Inline Cryptography
No Crypto Assist chip
Crypto locks core
True run-to-completion

DDR3 Control Plane 4xPCIe Data Plane 4xSGMI

DRAM (4 cores) (10 core)
BQS onFPGE
a core
One Core dedicated to BQS
1 Control Plane Core Ctrl SVC1 PPE1 PPE2 PPE3 PPE4 PPE5 Always active
RP and FECP-like roles DDR3(5+1 or 9+1 cores)
DRAM
SVC2 SVC3 PPE6 PPE7 PPE8 PPE9 PPE10

3 Services Core
No hardware TCAM
10 Gbps XAUI

System 1xSGMI
FPGA DSP
Mgmt Ethernet Multi Gigabit
Fabric
Console / Aux 10 Gbps/slot
Peripheral SM-X
Interconnect SM-X
USB
2Gb/slot
Flash

NIM
NIM
NIM

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
4351 Hardware Diagram (aka Utah)

Rangeley CPU mSATA

DRAM
(MO-300)

PPE1 PPE2 PPE3 PPE4 PPE5 Mgmt Ethernet Console, Aux & USB
Front Panel Ethernet System Glue Logic Console
Front Panel Ethernet
Front Panel Ethernet FPGA
PPE6 PPE7 PPE8
I2C to Modules
SPI Flash GE Switch

USB Host Ports PCIe Switch

eMMc

USB-to-SD
NIM Slots x 2 NGSM Slots x 2
NIM Slots x 2 NGSM slots x 2

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
4351 Hardware Diagram (aka Utah)
8 Cores @ 2.4 Ghz / 1 thread per core
1 core for RP/IOSd
1 core acting for Crypto & QoS
4 cores @ 1 thread/core for features
2 service cores

1 core
Rangeley as
CPU RP hosting IOSd mSATA
DRAM
(MO-300)

PPE1 PPE2 PPE3 PPE4 PPE5 2 service cores

Mgmt Ethernet Console, Aux & USB
Front Panel Ethernet System Glue Logic Console
Front Panel Ethernet
Front Panel Ethernet FPGA
PPE6 PPE7 PPE8
I2C to Modules
SPI Flash 1 core as Crypto and BQS GE Switch
2 cores QFP
2 cores QFP license activated

USB Host Ports PCIe Switch

eMMc

USB-to-SD
NIM Slots x 2 NGSM Slots x 2 4331 and 4321 are similar; just less cores
NIM Slots x 2 NGSM slots x 2
and expansion slots

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
 ESP
FECP

Generic ESP Block Diagram

QFP
Crypto
Assist. PPE BQS

intercon.

Reset / Pwr Ctrl Packet Buffer Part Len / BW

TCAM Resource DRAM
DRAM SRAM
Temp Sensor

EEPROM
QFP Complex
DDRAM Packet Processor Engine BQS

Boot Flash
(OBFL,) FECP PPE1 PPE2 PPE3 PPE4 PPE5

JTAG Ctrl PPE6 PPE7 PPE8 PPEN

Dispatcher
GE, 1Gbps
Packet Buffer
I 2C
SPA Control
Crypto SPA Bus
SPI Mux
Reset / Pwr Ctrl ESI, 11.2Gbps
SA table SPA-SPI, 11.2Gbps
DRAM Interconnect Hypertransport, 10Gbps
Other

RPs RPs ESP RPs SIPs

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
 Acronyms
RP Route Processor
FP Forwarding Processor = ESP (Embedded Service Processor)
CPP Cisco Packet Processor Compex= QFP (Quantum Flow Processor)
PPE Packet Processing Engine
IOCP I/O Control Processor
FECP Forwarding Engine Control Processor
SPA Shared Port Adapter
SIP SPA Interface Processor
IOSd IOS image that runs as a process on the RP
FMAN Forwarding manager (FMAN-RP, FMAN-FP)
Scbac FW Session Control Block
EOBC = Ethernet Out of Band Channels Packet Interface for Card to Card Control Traffic
IOS-XE (BinOS) = Linux Based Software Infrastructure That Executes on MCP

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Software Architecture
ASR1K Software Architecture
RP
CPU
RP
Chassis Manager
IOS CPU

ESI (10-40 Gbps)

Forwarding Manager

interconn.Linux GE switch
Kernel

ESP ESP
FECP Chassis Manager
EOBC (1 Gbps) FECP
Drivers Forwarding Manager
Drivers
I2C Drivers
Linux Kernel
Crypto
QFP
Assist.
QFP
Crypto
BQS
Assist.
interconn.
ESI (10-40 Gbps)

SIP SIP
IOCP interconn.
SPA Driver Chassis
SPA Driver
SPA Driver Manager
SPA
IOCP
Aggreg.
Linux Kernel

SPA SPA SPA SPA SPA

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Forwarding Manager (FMAN)
RP
CPU FMAN-RP
FMAN on RP communicates with
Chassis Manager FMAN process on ESP
IOS
Distributed function
ESI (10-40 Gbps)
Forwarding Manager

Linux Kernel
Propagates control plane ops. to ESP
FMAN-FP CEF tables, ACLs, NAT, SAs,
ESP FECP ESP Manager
Chassis aka Forwarding Plane
FMAN-FP communicates information
EOBC (1 Gbps)

Drivers
Drivers
Drivers
Forwarding Manager
back to FMAN-RP
I2C

Linux Kernel
e.g. statistics

QFP
Crypto
FMAN-RP pushes info back to IOS
BQS
Assist.
FMAN on active RP maintains state
ESI (10-40 Gbps)

SIP
IOCP
for both active & standby ESPs
SPA Driver Chassis
SPA Driver
SPA Driver Manager Facilitates NSF after re-start with bulk
download of state information
Linux Kernel

SPA SPA SPA

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
PPE Microcode
RP
Written in C
CPU
Chassis Manager
proper features, no hack
IOS
Runs on each thread of the PPE
ESI (10-40 Gbps)
Forwarding Manager

Linux Kernel

Processes packets
ESP FECP Chassis Manager run to completion
assisted by various memories
EOBC (1 Gbps)

Drivers Forwarding Manager

Drivers
PPE Microcode runs here
Drivers
TCAM, DRAM, various speeds
I2C

Linux Kernel

QFP
Features applied via FIA
QFP

Packet Processor Engine BQS
PPE PPE PPE PPE PPE
Crypto
BQS
1 2 3 4 5

PPE PPE PPE PPE
Assist.

6 7 8 N

Feature Invocation Array

Dispatcher
Packet Buffer
ESI (10-40 Gbps)

SIP
IOCP
SPA Driver
SPA Driver
SPA Driver
Chassis
Manager
FIA per interface
Linux Kernel
input FIA, output FIA
drop FIA (Null interface)
SPA SPA SPA

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
Resource Monitoring
The vital signs RP
CPU
Chassis Manager
IOS
Forwarding Manager
Control Plane CPUs
Linux Kernel

ESP FECP
Where does it hurt ?
Chassis Manager

Drivers Forwarding Manager

Drivers
Drivers
Linux Kernel

QFP
Data Plane CPUs Crypto
BQS
Assist.

SIP
IOCP
SPA Driver Chassis
SPA Driver
SPA Driver Manager

Linux Kernel

SPA SPA SPA

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
Example: IOS Memory Usage vs IOSd RP Utilization
asr-1k#show memory statistic
Load for five secs: 6%/1%; one minute: 5%; five minutes: 3% RP
Time source is NTP, 22:18:08.111 EDT Sat Apr 19 2014 CPU
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b) Chassis Manager
Processor 300AE008 1713127140 564269356 1148857784 1066242316 992444168 IOS
lsmpi_io 963791D0 6295088 6294120 968 968 968
Forwarding Manager
asr-1k#show process mem | inc BGP
523 0 2333028 51368 389076 313 313 BGP Router
Linux Kernel
asr-1k#show process cpu
Complex CLI, platform specific.

Additional information require connecting to the Linux shell

asr-1k#sh platform software process list RP active summary

Architecture : ppc
Memory (kB)
Physical : 4127744
Total : 3874516
Used : 2095636
Free : 1778880

asr-1k#sh platform software process list RP active | inc fman

fman_rp 29015 27992 29015 S 20 136847360

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
 QFP Memory Utilization
asr-1k#show platform hardware qfp active infrastructure exmem statistics
QFP exmem statistics

Type: Name: DRAM, QFP: 0

Total: 1073741824
It is getting worse InUse: 219466752
Free: 854275072
Lowest free water mark: 854005760
Type: Name: IRAM, QFP: 0
Total: 134217728
InUse: 8728576
Free: 125489152
Lowest free water mark: 125489152
Type: Name: SRAM, QFP: 0
Total: 32768
InUse: 15088
Free: 17680
Lowest free water mark: 17680

asr-1k#show platform hardware qfp active infrastructure exmem statistics user

10 279092 284672 CEF

40 36441494 36458496 NAT
ESP FECP Chassis Manager

Drivers Forwarding Manager

Drivers
asr-1k#show platform hardware qfp active tcam resource-manager usage Drivers
Load for five secs: 0%/0%; one minute: 1%; five minutes: 1% Linux Kernel
Time source is NTP, 09:43:55.075 EDT Fri Apr 25 2014

QFP TCAM Usage Information

QFP

<snip>
Crypto
BQS
Assist.
Total TCAM Cell Usage Information
----------------------------------
Name : TCAM #0 on CPP #0
Total number of regions : 3
Total tcam used cell entries
Total tcam free cell entries
:
:
28
524260
TCAM DRAM DRAM
Threshold status : below critical limit
LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
 IOS 3.14

Resources: simplified view RP

CPU
Chassis Manager
asr-1k# show platform resources IOS
Resource Usage Max Warning Critical State Forwarding Manager
RP0(ok, active) H
Control Processor 5.80% 100% 90% 95% H
Linux Kernel
DRAM 1814MB 3783MB 90% 95% H
ESP0(ok, active) H
Control Processor 19.89% 100% 90% 95% H
DRAM 683MB 1962MB 90% 95% H
QFP H ESP FECP Chassis Manager
DRAM 76244KB 524288KB 80% 90% H
IRAM 8817KB 131072KB 80% 90% H
SRAM 14KB 32KB 80% 90% H Drivers Forwarding Manager
Drivers
TCAM 28cells 131072cells 80% 90% H Drivers
ESP1(ok, standby) H Linux Kernel
Control Processor 19.89% 100% 90% 95% H
DRAM 683MB 1962MB 90% 95% H
QFP H QFP
DRAM 76244KB 524288KB 80% 90% H
Crypto
BQS
IRAM 8817KB 131072KB 80% 90% H Assist.
SRAM 14KB 32KB 80% 90% H
TCAM 28cells 131072cells 80% 90% H
SIP0 H
SIP
Control Processor 4.10% 100% 90% 95% H IOCP
DRAM 307MB 460MB 90% 95% H SPA Driver Chassis
SPA Driver
SIP1 H SPA Driver Manager
Control Processor 1.10% 100% 90% 95% H
DRAM 160MB 460MB 90% 95% H
Linux Kernel
**State Acronym: H - Healthy, W - Warning, C Critical
SPA SPA SPA

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
 Other show commands improved too
IOS 3.14

Improves interaction with TAC

show memory

show processes memory RP

CPU
Chassis Manager
show processes cpu IOS
Forwarding Manager

Linux Kernel

show memory platform

show processes memory platform

show processes cpu platform

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
Lab Access
Lab Portal on the Home Page in Firefox
Register with your email address
Class Name is wzhang_v23066
Once in the lab portal, click on the "Lab Access RDP Server"
icon from the Topology tab
Login with username: Flexvpn password: lp22#$BPA
Follow instructions provided by the lab instructor

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
 IOS-XE Troubleshooting Lab Topology 10.58.100.1/32
10.10.10.10/32
Router1

20.20.20.20/32
Router3
Gig 0/1 .1.5 .2
ISP-1 Gig 0/1 Gig0/1 .3.3
Router5
10.1.x.x/16

ISP-3
Gig 4

.1.1 CSR2

10.3.x.x/16
Router2
Gig 4

ISP-2 .3.1
Router4

172.16.2.x /30

Client Server
10.1.1.100 10.3.3.100

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Day in Life of Normal Packet
 SIP intercon.

SPA
IOCP
Aggreg.

Ingress Packet Through SIP ESPs

SPA SPA

Reset / Pwr Ctrl Interconnect

EV-RP
Temp Sensor EV-FC In ref
clocks
EEPROM

DDRAM Egress
Ingress
Buffer
Boot Flash IOCP Scheduler Status
(OBFL,) (SC854x SOC)

JTAG Ctrl
SPA Aggregation Network
clock
ASIC (Marmot) distribution

Ingress buffers Egress buffers

(per port) (per port)
Network
Ingress clocks

Reset / Pwr Ctrl SPA Agg.

Classifier C2W

SPA

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
 ESP
FECP

Ingress Packet Through ESP

QFP
Crypto
Assist. PPE BQS

intercon.

Reset / Pwr Ctrl Packet Buffer Part Len / BW

TCAM Resource DRAM
DRAM SRAM
Temp Sensor

EEPROM
QFP Complex
DDRAM Packet Processor Engine BQS

Boot Flash
(OBFL,) FECP PPE1 PPE2 PPE3 PPE4 PPE5

JTAG Ctrl PPE6 PPE7 PPE8 PPEN

Dispatcher
Packet Buffer

Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

RPs RPs ESP RPs SIPs

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
 ESP
FECP

Packet Dispatched to PPE Core

QFP
Crypto
Assist. PPE BQS

intercon.

Reset / Pwr Ctrl Packet Buffer Part Len / BW

TCAM Resource DRAM
DRAM SRAM
Temp Sensor

EEPROM
QFP Complex
DDRAM
PPE2
Packet Processor Engine BQS

Boot Flash
FECP PPE1 PPE2 PPE3 PPE4 PPE5

Thread 1

Thread 2

Thread 3

Thread 4
(OBFL,)

JTAG Ctrl PPE6 PPE7 PPE8 PPEN

Dispatcher
Packet Buffer

Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

RPs RPs ESP RPs SIPs

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
 ESP
FECP

Packet Dispatched to PPE Thread

QFP
Crypto
Assist. PPE BQS

intercon.

Reset / Pwr Ctrl Packet Buffer Part Len / BW

TCAM Resource DRAM
DRAM SRAM
Temp Sensor

EEPROM
QFP
DDRAM
PPE2
Packet Processor EngineComplex BQS

Boot Flash
FECP PPE1 PPE2 PPE3 PPE4 PPE5

Thread 1

Thread 2

Thread 3

Thread 4
(OBFL,)

JTAG Ctrl PPE6 PPE7 PPE8 PPEN

Dispatcher
Packet Buffer

Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

RPs RPs ESP RPs SIPs

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
 ESP
FECP

FIAs Applied on Packet by PPE Thread Crypto

QFP
Assist. PPE BQS

intercon.

X-ConnectReset / Pwr CtrlMPLS IPv4 IPv6 Packet Buffer Part Len / BW

TCAM Resource DRAM
DRAM SRAM
Temp Sensor

EEPROM Input FIA Output FIA

QFP
DDRAM
Netflow

Input ACL
PPE2
Packet Processor EngineNetflow Complex BQS

NAT
Boot Flash NBAR Classify
FECP PPE1 PPE2 PPE3 PPE4 PPE5

Thread 1

Thread 2

Thread 3

Thread 4
(OBFL,) NBAR Classify
MQC Classify IP Unicast
JTAG Ctrl PPE6 PPE7 PPE8 N
PPE

NAT
IP Multicast MQC Policing
PBR MAC Accounting
Dispatcher
Dialer IDLE Rst Packet For
Packet Buffer Output ACL
Us PPE2
URD
Crypto Thread 3
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

RPs RPs ESP RPs SIPs

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
 ESP
FECP

FIAs Applied on Packet by PPE Thread Crypto

Assist. PPE
QFP
BQS

intercon.

X-ConnectReset / Pwr CtrlMPLS IPv4 IPv6 Packet Buffer Part Len / BW

TCAM Resource DRAM
DRAM SRAM
Temp Sensor

EEPROM Input FIA Output FIA

QFP
DDRAM
Netflow

Input ACL
PPE2
Packet Processor EngineNetflow Complex BQS

NAT
Boot Flash NBAR Classify
FECP PPE1 PPE2 PPE3 PPE4 PPE5

Thread 1

Thread 2

Thread 3

Thread 4
(OBFL,) NBAR Classify
MQC Classify IP Unicast
JTAG Ctrl PPE6 PPE7 PPE8 N
PPE

NAT IP Multicast MQC Policing

PBR MAC Accounting

Dispatcher
Dialer IDLE Rst Packet For
Packet Buffer Output ACL
Us PPE2
URD
Crypto Thread 3
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

RPs RPs ESP RPs SIPs

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
 ESP
FECP

Leaving the PPE Thread Crypto

QFP
Assist. PPE BQS

intercon.

X-ConnectReset / Pwr CtrlMPLS IPv4 IPv6 Packet Buffer Part Len / BW

TCAM Resource DRAM
DRAM SRAM
Temp Sensor

EEPROM Input FIA Output FIA

QFP
DDRAM
Netflow

Input ACL
PPE2
Packet Processor EngineNetflow Complex BQS

NAT
Boot Flash NBAR Classify
FECP PPE1 PPE2 PPE3 PPE4 PPE5

Thread 1

Thread 2

Thread 3

Thread 4
(OBFL,) NBAR Classify
MQC Classify IP Unicast
JTAG Ctrl PPE6 PPE7 PPE8 N
PPE

NAT
IP Multicast MQC Policing
PBR MAC Accounting
Dispatcher
Dialer IDLE Rst Packet For
Packet Buffer Output ACL
Us PPE2
URD
Crypto Thread 3
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

RPs RPs ESP RPs SIPs

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
 ESP
FECP

Packet proceeding to BQS then SIP

QFP
Crypto
Assist. PPE BQS

intercon.

Reset / Pwr Ctrl Packet Buffer Part Len / BW

TCAM Resource DRAM
DRAM SRAM
Temp Sensor

EEPROM
QFP
DDRAM Packet Processor EngineComplex BQS

Boot Flash
(OBFL,) FECP PPE1 PPE2 PPE3 PPE4 PPE5

JTAG Ctrl PPE6 PPE7 PPE8 PPEN

Dispatcher
Packet Buffer

Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

RPs RPs ESP RPs SIPs

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
 SIP intercon.

SPA
IOCP
Aggreg.

Egress Packet Through SIP ESPs

SPA SPA

Reset / Pwr Ctrl Interconnect

EV-RP
Temp Sensor EV-FC In ref
clocks
EEPROM

DDRAM Egress
Ingress
Buffer
Boot Flash IOCP Scheduler Status
(OBFL,) (SC854x SOC)
JTAG Ctrl
SPA Aggregation Network
clock
ASIC (Marmot) distribution

Ingress buffers Egress buffers

(per port) (per port)
Network
Ingress clocks

Reset / Pwr Ctrl SPA Agg.

Classifier C2W

SPA

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Punt Path: From QFP to Internal Destination
ESP
RP
FECP
CPU

QFP
Crypto interconn. GE switch
Assist. PPE BQS

interconn.

Midplane

SIP interconn.

SPA
IOCP
Aggreg.

SPA SPA

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Punt Path: From QFP to Internal Destination
ESP PPE2
RP
FECP Thread 3 CPU

QFP
Crypto Punt to Recycle interconn. GE switch
Assist. PPE BQS
Punt to RP for us control
interconn. Punt to RP for us data
Punt to RP cause X Recycle path interface name on QFP: internal0/0/recycle:0

Midplane

SIP interconn. internal0/0/rp:0

SPA
IOCP
Aggreg. RP has is own dedicated internal interface on QFP: internal0/0/rp:0

SPA SPA

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
Inject Path: From RP via QFP to the network
ESP
RP
FECP
CPU

QFP
Crypto interconn. GE switch
Assist. PPE BQS

interconn.

Midplane

SIP interconn.

SPA
IOCP
Aggreg.

SPA SPA

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
Inject Path: Recycling packet via QFP to the network
ESP PPE2
RP
FECP Thread 3 CPU

QFP
Crypto Recycle path interconn. GE switch
Assist. PPE BQS

interconn.

Midplane

SIP interconn.

SPA
IOCP
Aggreg.

SPA SPA

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
Packet-tracer and FIA Debugger
The Packet Tracer and FIA Debugger IOS 3.10

X-ConnectReset / Pwr CtrlMPLS IPv4 IPv6

TCAM Resource DRAM
Packet Buffer Part Len / BW Packet # 16
Condition determines
Temp Sensor
DRAM SRAM

packets to be traced Output FIA Input ACL

Input FIA
EEPROM Optionally match
QFP
DDRAM
Pak Match ?
PPE2
Packet Processor EngineComplex
on the egress FIA
BQS
MQC Classify

Output ACL NAT

Boot Flash Input ACL
FECP PPE1 PPE2 PPE3 PPE4 PPE5
PBR

Thread 1

Thread 2

Thread 3

Thread 4
(OBFL,)
MQC Classify NAT
JTAG Ctrl PPE6 PPE7 PPE8 PPEN
NAT Encaps Output ACL
IP Unicast
PBR Statistics and final action will be NAT
Dispatcher Crypto
collected (matched packets dropped,
Packet Buffer punted to RP, forwarded to output Encaps
interface )PPE2
Crypto Thread 3 Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect
Optionally, FIA actions can logged per packet
System can capture several packets flows
RPs RPs ESP RPs SIPs Packet flows can be reviewed in show commands

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
Packet-Trace: Accounting
Accounting keeps a count of all pactrac interesting packets that enter and leave
the packet processor. There are three basic count groups.
Summary counts
Packets Matched packets that matched conditions
Packets Traced packets that were traced
Arrival counts
Ingress packets entering via external interfaces
Inject* number of packets seen as injected from control plane

2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet-Trace: Accounting
Departure counts
Forward number of packets scheduled/queued for delivery
Punt* number of packets punted to control plane
Drop* number of packets specifically dropped by packet processing
Consume number of packets consumed during packet process (e.g. ping request)

* Per reason/code counts are maintained for Inject, Punt and Drop.

2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet-Trace: Summary Data
When enabled, summary data is collected for a specified number of packets and
includes:
Packet number (pactrac specific packet number)
Input interface
Output interface
Final packet state and any punt/drop/inject codes
Collecting summary data uses little performance over the normal packet
processing. An example usage may be to isolate which interfaces are dropping
traffic so more detailed inspection can be used after applying interface specific
conditions.

2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet-Trace: Path Data
Path data may be collected per packet for a limited number of packets and is
made up of different types of data as follows:
Common path data (e.g. IP tuple)
Feature specific data (e.g. NAT)
Feature Invocation Array (FIA) trace optionally enabled
Copy of all or part of the incoming and/or outgoing packet optionally enabled
Capturing path data has the greatest impact on packet processing* capability
specifically FIA trace and packet copy.
FIA tracing creates many path data entries costing instructions and DRAM writes
Packet copy creates many DRAM read/writes
*Recall the packet-trace will only affect the performance of packets traced (i.e.
those matched by the user provided conditions)

2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Conditionally Matching Packets
Identifying Interesting Packets
asr-1k# debug platform condition ?
debug platform condition ?
both Simultaneous ingress and egress debug
egress Egress only debug

ingress Ingress only debug
interface Set interface for conditional debug
ipv4 Debug IPv4 conditions
ipv6 Debug IPv6 conditions
mpls Debug MPLS conditions

Match all ingress packets

asr-1k#debug platform condition ingress
Match all ingress packets on interface gig0/0/3
asr-1k#debug platform condition interface gig0/0/3 ingress
asr-1k#debug platform condition ipv4 10.0.0.1/32 both Match in & out packets with source or
asr-1k#debug platform condition ipv4 access-list 100 egress destination 10.0.0.1
asr-1k#debug platform condition mpls 10 1 ingress
Match egress packets passing
access-list 100
Match MPLS packets with
top ingress label 10
LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
Activating the Packet Tracer
Following packets through IOS-XE Basic Statistics
asr-1k# debug platform packet-trace ?
copy Copy packet data
The packet tracer follows a
drop Trace drops only set of packets in details
enable Enable packet trace through the FIA
packet Packet count

asr-1k# debug platform condition interface gig0/0/0 ingress

asr-1k# debug platform condition start
asr-1k# debug platform packet-trace enable
asr-1k# !send traffic Extraneous command -
asr-1k# show platform packet-trace statistics was suppressed in 16.3
Packets Summary
Matched 102 102 packets were matched by
Traced 0 the condition
Packets Received
Ingress 12
Inject 90
Count Code Cause
90 9 QFP ICMP generated packet
Packets Processed
Forward 12 12 packets were forwarded
Punt 0
Drop 90 90 packets were dropped
Count Code Cause
13 92 Ipv4Null0 13 packets were dropped
17 47 FirewallInvalidZone due to no route
60 184 FirewallL4
Consume 0 17 packets were dropped due to
60 packets dropped by L4 inspection absence of zone pair
(e.g. receiving window) LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Packet Tracer Tracing Packets
The fate of 16 packets
asr-1k# debug platform condition interface gig0/0/0 ingress
asr-1k# debug platform condition start Automatically stops tracing
asr-1k# debug platform packet-trace packet 16 after 16 packets
asr-1k# debug platform packet-trace enable Extraneous command -
asr-1k# !send traffic was suppressed in 16.3
asr-1k# show platform packet-trace summary
Pkt Input Output State Reason 16 packets were traced; we
0 Gi0/0/2 internal0/0/rp:0 PUNT 55 (For-us control) can zoom in
1 Gi0/0/2 internal0/0/rp:0 PUNT 55 (For-us control)
2 Gi0/0/2 internal0/0/rp:0 PUNT 55 (For-us control)
3 Gi0/0/2 internal0/0/rp:0 PUNT 55 (For-us control)
4 INJ.7 Gi0/0/2 FWD
5 INJ.7 Gi0/0/2 FWD
6 Gi0/0/2 internal0/0/rp:0 PUNT 55 (For-us control)
7 INJ.7 Gi0/0/2 FWD INJ.7: Packet injected by the RP
internal0/0/rp:0: Packet punted to the RP
8

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
Packet Tracer Tracing Packets
The fate of an individual packet
asr-1k# show platform packet-trace packet 1
Packet: 1 CBUG ID: 109056985
Summary
Zooming on packet 1
Input : GigabitEthernet0/0/2
Output : internal0/0/rp:0
State : PUNT 55 (For-us control)
Timestamp
Start : 334771580191282 ns (04/29/2014 08:01:38.017738 UTC)
Stop : 334771580487612 ns (04/29/2014 08:01:38.018035 UTC)
Path Trace
Feature: IPV4
Source : 17.0.0.196 Feature specific details are
Destination : 172.18.0.1 displayed
Protocol : 50 (ESP)
Feature: IPSec Only major features
Action : DECRYPT are shown
SA Handle : 753
SPI : 0x30ba5940
Peer Addr : 17.0.0.196
Local Addr: 172.18.0.1
LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
Packet Tracer Tracing Packets
... even keeping a copy of the packet if necessary
asr-1k# debug platform condition interface gig0/0/0 ingress
Keep a copy of the packet in
asr-1k# debug platform condition start ingress and egress of the ESP
asr-1k# debug platform packet-trace packet 16 (before and after the FIA)
asr-1k# debug platform packet-trace copy packet both [l2 | l3 | l4]
asr-1k# debug platform packet-trace enable
asr-1k# !send traffic Can store L2, L3 or L4
asr-1k# show platform packet-trace packet 1 pick-a-choose
Packet: 1 CBUG ID: 109056985
Summary
Display the stored packet copy
Input : GigabitEthernet0/0/2
Output : internal0/0/rp:0
State : PUNT 55 (For-us control)
Path Trace
Feature: IPV4
Feature: IPSec
Packet Copy In
45c00088 c5ee0000 ff32346f 11000313 ac120001 d4b46317 0000017c 68a60265
0ef58135 650e2341 15cf6e81 dd434455 b42efef8 c6cf5ab1 44ad3f98 b165c3d5
Packet Copy Out
45c0003c 00000000 015804f4 c0ab1301 e000000a 0205efc8 00000000 00000000
00000000 0000000a 0001000c 01000100 0000000f 00040008 0a000200

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
Packet Tracer Tracing Packets
The fate of a single packet even more more more details
asr-1k# show platform packet-trace packet 1 decode
Packet: 1 CBUG ID: 109056985
Summary
Input : GigabitEthernet0/0/2 Decode the stored packet copy
Output : internal0/0/rp:0
State : PUNT 55 (For-us control)
Path Trace
Feature: IPV4
Feature: IPSec
Packet Copy In
45c00088 c5ee0000 ff32346f 11000313 ac120001 d4b46317 0000017c 68a60265
0ef58135 650e2341 15cf6e81 dd434455 b42efef8 c6cf5ab1 44ad3f98 b165c3d5
IPv4
Version : 4
Header Length : 5
ToS : 0xc0
Total Length : 136
Identifier : 0xc5ee
IP Flags : 0x0
Frag Offset : 0 Here showing the input copy
TTL : 255 (output copy follows)
Protocol : 50 (ESP)
Header Checksum : 0x346f
Source Address : 17.0.3.19
Destination Address : 172.18.0.1
ESP
SPI : 0xd4b46317
Sequence Number : 0x0000017c
...

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
Packet Tracer Focus on Drops
Dropped packets nothing else
For drops, condition is optional
asr-1k# debug platform condition interface gig0/0/0 ingress
asr-1k# debug platform condition start
asr-1k# debug platform packet-trace packet 16 Only save dropped packets
asr-1k# debug platform packet-trace drop [code <dropcode>]
asr-1k# debug platform packet-trace enable Focus on specific drop codes
asr-1k# !send traffic (find codes in packet-trace statistics)
asr-1k# debug platform condition stop Stop tracing before dumping the
asr-1k# show platform packet-trace summary summary (code limitation)
Pkt Input Output State Reason
0 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput)
Admire dropped packets real close
1 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput)
asr-1k#show platform packet-trace packet 1
2 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput) Packet: 1 CBUG ID: 148787639
Summary
3 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput) Input : GigabitEthernet0/0/2
Output : GigabitEthernet0/0/2
4 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput) State : DROP 53 (IpsecInput)
5 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput) Timestamp
Start : 361426338620013 ns (04/29/2014 15:25:52.785406 UTC)
6 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput) Stop : 361426338684993 ns (04/29/2014 15:25:52.785471 UTC)
Path Trace
7 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput) Feature: IPV4
Source : 17.0.1.34
8 Destination : 172.18.0.1
Protocol : 50 (ESP)
Packet Copy Out
002304bb 72020007 7dfbe301 080045c0 0088d135 0000fe32 2c191100 0122ac12
0001085e 1d620000 00c8172c e8010c3e 44726e6f 3eb231d5 166298c1 f519313c
57
LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Tracing Basic and FIA-TRACE
Features Pack Tracer Pack Tracer w/ FIA-TRACE
asr1000# show platform hardware qfp active interface if-name asr1000#show platform packet-trace packet 1 asr1000#show platform packet-trace packet 0
gig1 Packet: 1 CBUG ID: 518 Packet: 0 CBUG ID: 655
General interface information Summary Summary
Interface Name: GigabitEthernet1 Input : GigabitEthernet1 Input : GigabitEthernet1
Interface state: VALID Output : GigabitEthernet3 Output : GigabitEthernet3
Platform interface handle: 7 State : FWD State : FWD
QFP interface handle: 6 Timestamp Timestamp
Start : 5331698002827 ns (07/11/2016 23:28:23.187027 UTC) Start : 5456699323393 ns (07/11/2016 23:30:28.244810 UTC)
Protocol 0 - ipv4_input Stop : 5331698159842 ns (07/11/2016 23:28:23.187184 Stop : 5456699556099 ns (07/11/2016 23:30:28.245043 UTC)
FIA handle - CP:0x2fccfe0 DP:0xe73998c0 UTC) Path Trace
[] Path Trace

IPV4_INPUT_DST_LOOKUP_ISSUE (M)
IPV4_INPUT_ARL_SANITY (M)
CBUG_INPUT_FIA

Feature: IPV4
DEBUG_COND_INPUT_PKT Feature: IPV4 Input : GigabitEthernet1
Input : GigabitEthernet1 Output : <unknown>
Output : <unknown> Source : 192.168.3.1
Source : 192.168.3.1 Destination : 192.168.255.167
Destination : 192.168.255.167 Protocol : 50 (ESP)
Protocol : 50 (ESP) Feature: FIA_TRACE
Input : GigabitEthernet1
Output : <unknown>
Entry : 0x8139f260 - DEBUG_COND_INPUT_PKT
Lapsed time : 9680 ns

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
Packet Tracing Basic and FIA-TRACE (II)
Features Pack Tracer Pack Tracer w/ FIA-TRACE

IPV4_INPUT_DST_LOOKUP_CONSUME (M) Feature: FIA_TRACE

Input : GigabitEthernet1
Output : <unknown>
Entry : 0x813a5554 -
IPV4_INPUT_DST_LOOKUP_CONSUME
IPV4_INPUT_ACL Lapsed time : 9320 ns

Feature: FIA_TRACE
Input : GigabitEthernet1
Output : <unknown>
Entry : 0x80f67140 - IPV4_INPUT_ACL
IPV4_INPUT_FOR_US_MARTIAN (M) Lapsed time : 60613 ns

Feature: FIA_TRACE
Input : GigabitEthernet1
Output : <unknown>
Entry : 0x813a5558 - IPV4_INPUT_FOR_US_MARTIAN
Lapsed time : 303133 ns

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
Packet Tracing Basic and FIA-TRACE (III)
Features Pack Tracer Pack Tracer w/ FIA-TRACE

IPV4_INPUT_STILE_LEGACY Feature: CFT Feature: CFT

API : cft_handle_pkt API : cft_handle_pkt
packet capabilities : 0x0000008c packet capabilities : 0x0000008c
input vrf_idx :0 input vrf_idx :0
calling feature : STILE calling feature : STILE
direction : Input direction : Input
triplet.vrf_idx :0 triplet.vrf_idx :0
triplet.network_start : 0x00000000 triplet.network_start : 0x00000000
triplet.triplet_flags : 0x00000000 triplet.triplet_flags : 0x00000000
triplet.counter :0 triplet.counter :0
cft_bucket_number : 2120447 cft_bucket_number : 2120447
cft_l3_payload_size : 100 cft_l3_payload_size : 100
cft_pkt_ind_flags : 0x00000000 cft_pkt_ind_flags : 0x00000000
cft_pkt_ind_valid : 0x00000935 cft_pkt_ind_valid : 0x00000935
tuple.src_ip : 192.168.3.1 tuple.src_ip : 192.168.3.1
tuple.dst_ip : 192.168.255.167 tuple.dst_ip : 192.168.255.167
[] []
Feature: NBAR Feature: NBAR
Packet number in flow: N/A Packet number in flow: N/A
Classification state: Final Classification state: Final
Classification name: ipsec Classification name: ipsec
[] Classification ID: [CANA-L7:9]
Number of matched sub-classifications: 0
Number of extracted fields: 0
Is PA (split) packet: False
TPH-MQC bitmask value: 0x0

Feature: FIA_TRACE
Input : GigabitEthernet1
Output : <unknown>
Entry : 0x80fa0f88 - IPV4_INPUT_STILE_LEGACY
Lapsed time : 396533 ns
LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
Packet Tracing Basic and FIA-TRACE (IV)
Features Pack Tracer Pack Tracer w/ FIA-TRACE
IPV4_INPUT_QOS Feature: QOS Feature: QOS
Direction : Ingress Direction : Ingress
Action : SET Action : SET
Fields : DSCP Fields : DSCP
Feature: FIA_TRACE
Input : GigabitEthernet1
Output : <unknown>
Entry : 0x813a6fe4 - IPV4_INPUT_QOS
Lapsed time : 64586 ns

IPV4_INPUT_VFR Feature: FIA_TRACE

Input : GigabitEthernet1
Output : <unknown>
Entry : 0x813a5574 - IPV4_INPUT_VFR
Lapsed time : 3653 ns

IPV4_NAT_INPUT_FIA Feature: FIA_TRACE

Input : GigabitEthernet1
Output : <unknown>
Entry : 0x80f99600 - IPV4_NAT_INPUT_FIA
Lapsed time : 303560 ns

IPV4_INPUT_LOOKUP_PROCESS (M) Feature: FIA_TRACE

Input : GigabitEthernet1
Output : <unknown>
Entry : 0x813ae9b0 - IPV4_INPUT_LOOKUP_PROCESS
Lapsed time : 29306 ns

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
Packet Tracing Basic and FIA-TRACE (V)
Features Pack Tracer Pack Tracer w/ FIA-TRACE

IPV4_INPUT_IPOPTIONS_PROCESS (M) Feature: FIA_TRACE

Input : GigabitEthernet1
Output : <unknown>
Entry : 0x813a557c - IPV4_INPUT_IPOPTIONS_PROCESS
Lapsed time : 2813 ns

IPV4_INPUT_GOTO_OUTPUT_FEATURE (M) Feature: FIA_TRACE

Input : Virtual-Access19
Output : <unknown>
Entry : 0x813ae9b4 -
IPV4_INPUT_GOTO_OUTPUT_FEATURE

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
Debugging strategies
Everyday situations

Traffic did not reach its target !

What happened to that packet ?
Why did that happen ?

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Everyday situations
IPsec ZBF NAT
WAAS SNMP
OTV
First Routing

Which feature went wrong ?

What went wrong in the feature ?

Memory
Config Performance Ordering
Bug
Traffic Ambiguity
issue

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
Everyday situations Second

What went wrong in the feature ?

Performanc Memory
Config Performance Ordering
e
Bug
Traffic Ambiguity
issue

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
 Using statistics for troubleshooting packet drops
SPA
show interfaces <interface-name>
show interfaces <interface-name> accounting
show interfaces <interface-name> stats
SIP
show platform hardware port <slot/card/port> plim statistics
show platform hardware subslot {slot/card} plim statistics
show platform hardware slot {slot} plim statistics
show platform hardware slot {0|1|2} plim status internal
show platform hardware slot {0|1|2} serdes statistics
RP
show platform hardware slot {r0|r1} serdes statistics
show platform software infrastructure lsmpi
Not easy not very practical either.
ESP Lets dig deeper before making it simpler
show platform hardware slot {f0|f1} serdes statistics
show platform hardware slot {f0|f1} serdes statistics internal
show platform hardware qfp active bqs 0 ipm mapping
show platform hardware qfp active bqs 0 ipm statistics channel all
show platform hardware qfp active bqs 0 opm mapping
show platform hardware qfp active bqs 0 opm statistics channel all
show platform hardware qfp active statistics drop [detail]
show platform hardware qfp active interface if-name <Interface-name> statistics
show platform hardware qfp active infrastructure punt statistics type per-cause |
exclude _0_
show platform hardware qfp active infrastructure punt statistics type punt-drop |
exclude _0_
show platform hardware qfp active infrastructure punt statistics type inject-drop |
exclude _0_
show platform hardware qfp active infrastructure punt statistics type global-drop |
exclude _0_
show platform hardware qfp active infrastructure bqs queue output default all
show platform hardware qfp active infrastructure bqs queue output recycle all
LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
Debugging Strategies to Date

Top Down
IOS Control Plane
ACL + show access-list,
show interface / ip route / bgp
Rock bottom

Platform Control Plane

ESP stuff Lets

e.g. show platform hard to remember change
that!!
Data Plane
ESP stuff
More arcane show platform

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
Troubleshooting Tools and
Capabilities
Understanding and Extracting ESP
Logs
ESP Tracing aka Logging
TEMP RAM FS
RP RP logs are first written
CPU
Chassis Manager
here (efficiency)
IOS
NFS Shared Disk

ESI (10-40 Gbps)

Forwarding Manager
Hard disk is really here
Linux Kernel

ESP FECP TEMP RAM FS

Chassis Manager
ESP logs are first written
EOBC (1 Gbps)
Drivers Forwarding Manager
here (efficiency)
Drivers
I2C Drivers
Linux Kernel

QFP Mounted NFS

Crypto
BQS
Assist. ESP logs are committed

here at regular intervals
ESI (10-40 Gbps)

SIP
IOCP
SPA Driver Chassis
SPA Driver
SPA Driver Manager

Linux Kernel

SPA SPA SPA

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Important logs
RP
CPU
Chassis Manager
fman_rp_R[0|1]-0.log
IOS
Under /harddisk/tracelogs

ESI (10-40 Gbps)

Forwarding Manager
fman_rp_R[0|1]-0.log.<timestamp>
Linux Kernel
fman-fp_R0.log.<timestamp>
cpp_cp_F[0|1]-0.log.<timestamp>
ESP FECP Chassis Manager
EOBC (1 Gbps) fman_fp_F[0|1]-0.log
Drivers Forwarding Manager cpp_cp_F[0|1]-0.log
Drivers
I2C Drivers
Linux Kernel

QFP Under /harddisk/tracelogs/

Crypto fman-fp_R0.log.<timestamp>
BQS
Assist.
cpp_cp_F[0|1]-0.log.<timestamp>
ESI (10-40 Gbps)

SIP
IOCP
SPA Driver Chassis
SPA Driver
SPA Driver Manager

Linux Kernel

SPA SPA SPA

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
What log files are important?
Important log files to get for security issues:
fman_rp_R[0|1].log (under /tmp/rp/trace directory on RP)
fman-fp_F[0|1]-0.log (under /tmp/fp/trace directory on ESP
cpp_cp_F[0|1]-0.log (under /tmp/fp/trace directory on ESP)

All these logs get rotated and are copied to /harddisk/tracelogs directory on
active RP.
Look for the relevant log files depending on the time of the failure
By default, all ERR messages are logged should be the first things to look for

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
 Example log files

The timestamp

My-ASR1000-2#dir harddisk:/tracelogs/cpp_cp_F0*
Directory of harddisk:/tracelogs/cpp_cp_F0*
Directory of harddisk:/tracelogs/
3768365 -rwx 1048934 Jan 6 2014 18:20:16 +00:00 cpp_cp_F0-0.log.7133.20140106182015
3768330 -rwx 551643 Jan 7 2014 09:27:51 +00:00 cpp_cp_F0-0.log.7133.20140107092751
3768335 -rwx 1048901 Jan 7 2014 08:56:44 +00:00 cpp_cp_F0-
0.log.7133.2014010708564339313059840 bytes total (30680653824 bytes free)

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
 Rotating the log files

My-ASR1000-2#dir harddisk:/tracelogs/cpp_cp_F0*
Directory of harddisk:/tracelogs/cpp_cp_F0*
Directory of harddisk:/traceMy-ASR1000-2#test platform software trace slot rp active forwarding-manager rotate
Rotated file from: /tmp/rp/trace/stage/fman_rp_R0-0.log.13836.20140107094754, Bytes: 0, Messages: 6535
My-ASR1000-2#test platform software trace slot FP active cpp-control-process rotate
Rotated file from: /tmp/fp/trace/stage/cpp_cp_F0-0.log.7133.20140107093650, Bytes: 154027, Messages: 786
My-ASR1000-2#test platform software trace slot FP active forwarding-manager rotate
Rotated file from: /tmp/fp/trace/stage/fman-fp_F0-0.log.8247.20140107093738, Bytes: 20170, Messages: 210

OR use

My-ASR1000-2#request platform software trace rotate all

Does not show the rotated file names w/

time stamp have to hunt them down

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
Embedded Packet Capture
 The Embedded Packet Capture
IOS 3.7

One way of capturing packets

Device# monitor capture mycap start

Device# monitor capture mycap access-list v4acl
Device# monitor capture mycap limit duration 1000
Device# monitor capture mycap interface GigabitEthernet 0/0/1 both
Device# monitor capture mycap buffer circular size 10
Device# monitor capture mycap start Shows whether packets have been received or sent
Device# monitor capture mycap export tftp://10.1.88.9/mycap.pcap
Device# monitor capture mycap stop
Shows what packets look like
Requires hex dump analysis or export to decoder (sniffer)
Does not tell us what happened to the packet
Device# show monitor capture mycap buffer dump

0
0000: 01005E00 00020000 0C07AC1D 080045C0 ..^...........E.
0010: 00300000 00000111 CFDC091D 0002E000 .0..............
0020: 000207C1 07C1001C 802A0000 10030AFA .........*......
0030: 1D006369 73636F00 0000091D 0001 ..example....... Excellent tool but insufficient in many cases
1
0000: 01005E00 0002001B 2BF69280 080046C0 ..^.....+.....F.
0010: 00200000 00000102 44170000 0000E000 . ......D.......
0020: 00019404 00001700 E8FF0000 0000 .............. http://www.cisco.com/en/US/docs/ios-
xml/ios/epc/configuration/xe-3s/asr1000/nm-packet-capture-
2 xe.html
0000: 01005E00 0002001B 2BF68680 080045C0 ..^.....+.....E.
0010: 00300000 00000111 CFDB091D 0003E000 .0..............
0020: 000207C1 07C1001C 88B50000 08030A6E ...............n
0030: 1D006369 73636F00 0000091D 0001 ..example.......
LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
Embedded Packet Capture
Export
EPC added to FIA TFTP
Server
Beginning of ingress FIA
End of egress FIA Capture point Capture Buffer

Matched packets are copied Gi0/0/1 Gi0/0/2

Router
Copied packets get punted to RP
Original packets processed as usual
Capture buffer on RP can be exported
to .pcap file

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
Conditional Feature Debugging
The Packet Tracer and Conditional Debugger
IOS 3.10

X-ConnectReset / Pwr Ctrl

L2 Switch IPv4 IPv6 MPLSPart Len / BW
TCAM Resource DRAM
Packet Buffer Packet # 16
Condition determines
Temp Sensor
DRAM SRAM

packets to be traced Output FIA Input ACL

EEPROM Input FIA
QFP
DDRAM
Ingress Match ?
Input ACL PPE2
Packet Processor EngineComplex BQS
MQC Classify

Output ACL NAT

Boot Flash
FECP
MQC
PPE1 PPE2 PPE3 PPE4 PPE5

Thread 1

Thread 2

Thread 3

Thread 4
(OBFL,) PBR
Classify NAT
JTAG Ctrl PPE6 PPE7 PPE8 PPEN
Encaps Output ACL
NAT IP Unicast
NAT
Dispatcher Crypto
Packet Buffer Encaps
PBR If feature conditional debugger is PPE2
activated, these blocks will be
Crypto Thread 3 Crypto
SPI Mux debugged
Reset / Pwr Ctrl
SA table
DRAM Interconnect
The packet tracer collects statistics
and final action (matched packets
Our focus now dropped, punted to RP, forwarded to
RPs RPs ESP RPs SIPs output interface )

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
Conditionally Matching Packets
Step 1 Identifying packets
asr-1k# debug platform condition ?
debug platform condition ?
both Simultaneous ingress and egress debug
egress Egress only debug

ingress Ingress only debug
interface Set interface for conditional debug
ipv4 Debug IPv4 conditions
ipv6 Debug IPv6 conditions
mpls Debug MPLS conditions

Match all ingress packets

asr-1k#debug platform condition ingress
Match all ingress packets on interface gig0/0/3
asr-1k#debug platform condition interface gig0/0/3 ingress
asr-1k#debug platform condition ipv4 10.0.0.1/32 ingress Match ingress packets with source or
asr-1k#debug platform condition ipv4 access-list 100 ingress destination 10.0.0.1
asr-1k#debug platform condition mpls 10 1 ingress
Match ingress packets passing
access-list 100
Match MPLS packets with
top ingress label 10
LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
Feature Debugging
Step 2 Define feature(s) to troubleshoot

asr-1k# debug platform condition feature ?

acl ACL feature
alg ALG feature
fw FW feature
ipsec IPSEC feature
nat NAT feature Many features are supported but
nat64 NAT64 feature focus is on NAT, ZBF and FW at
the moment

acl ACL feature cxsc CXSC feature nat64 NAT64 feature

alg ALG feature dpss DPSS feature nbar NBAR feature
appnav AppNav feature evc EVC feature overlay overlay feature
atm ATM feature fw FW feature qos QOS feature
atom ATOM feature ipsec IPSEC feature subscriber Subscriber feature
bridge-domain Layer2 bridging feature lisp LISP feature tcp TCP feature
cent CENT feature multicast multicast feature vpls VPLS feature
cft CFT feature nat NAT feature

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
Feature Debugging (cont.)
Step 2 (cont.) Define feature submodes to be troubleshot

asr-1k# debug platform condition feature fw dataplane submode ?

alg-inspect Debug firewall ALG inspect information
all Debug firewall all information
detail Debug firewall detail
drop Debug firewall drop information
event Debug firewall event information
ha Debug firewall HA information
layer4 Debug firewall Layer 4 information
level Debug level information
policy Debug firewall policy information

asr-1k# debug platform condition feature fw dataplane submode drop layer4 policy

Multiple submodes can be active at once

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
Start & Stop Conditional Debugging
Step 3 Start marking the packets (internally) and debug features

asr-1k# debug platform condition ?

start Start conditional debug
stop Stop conditional debug

asr-1k#debug platform condition start Debugs wont show on

asr-1k#... console (yet)
asr-1k#debug platform condition stop

After this, analyze the debugs

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
Wrapping up
Please join us for the Service Provider Innovation Talk featuring:
Yvette Kanouff | Senior Vice President and General Manager, SP Business
Joe Cozzolino | Senior Vice President, Cisco Services

Thursday, July 14th, 2016

11:30 am - 12:30 pm, In the Oceanside A room

What to expect from this innovation talk

Insights on market trends and forecasts
Preview of key technologies and capabilities
Innovative demonstrations of the latest and greatest products
Better understanding of how Cisco can help you succeed

Register to attend the session live now or

watch the broadcast on cisco.com
Thank You
Complete Your Online Session Evaluation
Please complete your Online
Session Evaluations after each
session
Complete 4 Session Evaluations &
the Overall Conference Evaluation
(available from Thursday) to receive
your Cisco Live T-shirt
All surveys can be completed via
the Cisco Live Mobile App or the
Dont forget: Cisco Live sessions will be available
Communication Stations for viewing on-demand after the event at
CiscoLive.com/Online

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
Continue Your Education
Demos in the Cisco campus
Walk-in Self-Paced Labs
Lunch & Learn
Meet the Engineer 1:1 meetings
Related sessions

LTRARC-3500 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 89
Q&A
Thank You