2013 - Usa - PDF - BRKCRS-3142 - Troubleshooting Cisco Catalyst 4500 Series Switches

Troubleshooting Cisco Catalyst 4500 Series
Switches
BRKCRS-3142
2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3142 Cisco Public
Session Goals
At the end of this session, you should be able to:
Understand system resources and monitor their usage
Identify all areas of packet loss
Trace hardware packet path
Make use of newer tools

This content is based on questions we see in the field. Feedback is welcome!

3
Agenda
Products Overview
Troubleshooting
Method
System Resources
Packet path / loss
VSS
PoE
Netflow
Tools/Tips
Appendix
4
Products Overview
5 5
4503-E
4507R+E 4510R+E 4506-E
6 Gbps per slot
Classic supervisors
Classic line cards
e.g, SupV-10GE, 45xx line card
See the appendix for supervisor, line card, and chassis product and compatibility details.
48 Gbps per slot
+E Chassis support 12.2(53)SG4 onward
switch, Sup7L-E, 47xx line card
4507R+E, 4510R+E, 4503-E, 4506-E
24 Gbps per slot
-E Chassis support 12.2(31)SGA6 onward
Sup6-E, Sup6L-E and 46xx line card
4507R-E, 4510R-E
5
Products Overview
6
1. Ternary Content Addressable Memory
2. Optional for Supervisor IV and V. Integrated in Supervisor V-10GE, switch, 7L-E
Intelligent Supervisors
Supervisor Engine 7-E, 7L-E, 6-E, 6L-E, V-10GE, V, IV, II-Plus-10GE,
II-Plus-TS, II-Plus
Transparent Line Cards
Wire-rate, oversubscribed, PoE
10/100, 10/100/1000, GE, 10GE
Various physical media front panel ports
Dedicated per-slot bandwidth to supervisor
Switching ASICs
Packet Processor
Forwarding Engine
Specialized Hardware
TCAM
1
s for ACLs, QoS, L3 forwarding
NetFlow
2
(NFE) for statistics gathering
6
Shared Packet
Memory
Line Card
Stub ASICs
Front Panel Ports
Supervisor
NFE
2
CPU

TCAMs
1
Packet
Processor

Forwarding
Engine

6
Agenda
Products Overview
Troubleshooting
Method
System Resources
Packet path / loss
VSS
PoE
Netflow
Tools/Tips
Appendix
7
Troubleshooting Method
General Recommendations
Design with intent
ideally, create a deterministic network
engineers not traffic should control the network

Baseline, monitor against baseline, alarm and/or adjust
problems are solved faster when knowns can be eliminated

Characterize issues quickly with a plan
8
Method
1. Define Problem
2. Gather Facts
3. Consider Possibilities
4. Create Action Plan
5. Execute Action Plan
6. Observe Results
D
o
c
u
m
e
n
t
a
t
i
o
n

Symptoms? System Messages? User
Input?
When? Frequency? Impact? Scope?
Need to have a good understanding about
how the system looks like when it is healthy
Further information and examples are in
the troubleshooting section
Want to learn more? Check out CCNP Practical Studies: Troubleshooting by Donna Harrington.
CCNP TSHOOT 642-832 Official Certification Guide by Kevin Wallace.
9
Method
Category Possible Cause
Config/Design Mis-configuration
Reaching Capacity
Traffic DOS Attack
Traffic Pattern Change
Bad peer/server
Software Issue Software Limitation
Bug
Hardware Issue Hardware Limitation
Failed Hardware
Transient Hardware Issue
1. Define Problem
2. Gather Facts
6. Observe Results
D
o
c
u
m
e
n
t
a
t
i
o
n

10
Method
1. Define Problem
2. Gather Facts
6. Observe Results
D
o
c
u
m
e
n
t
a
t
i
o
n

What needs to be done to isolate each
potential root cause?
Make a change, measure results,
rollback change if problem persists
Problem solved? If not, continue
action plan
11
Before you dig deep
Top down approach
Hardware generally does what its told to do
Before you troubleshoot the platform, rule out the usual suspects

End-to-end Compare traffic at endpoints
Keep standard methods/tools for loss
measurement handy

Iperf
Security Port security issues
Actions are not always sent to syslog
Restrict modes may use CPU
802.1x, DAI, DHCP snooping/relay, IPSG, Port Security, PACL
Common
Issues
Security features
L2
L3 unicast
L3 multicast
RACL, VACL, unicast RPF, intermediary stateful inspection
spanning-tree topology, IGMP snooping
reachability, peer adjacency
rpf, L3 path construction (RP), IGMP groups
12
Caution
debug and show platform commands to follow
Excessive debug output to console may disable switch
show platform commands are intended for in-depth troubleshooting
Use debug and show platform commands only when advised by TAC
show platform CLIs are not officially supported IOS commands
Not all commands apply to all platforms.
Some are IOS-XE specific (Supervisor 7-E, 7L-E and 4500X)
13
Agenda
Products Overview
Troubleshooting
Method
System Resources
Packet path / loss
VSS
PoE
Netflow
Tools/Tips
Appendix
14
System Resources
CPU
Runs IOS/IOS-XE processes
Runs 4500 platform-specific processes
Sends/Receives control traffic
Software-switches packets that cant be hardware-switch

Elevated CPU = in-use CPU, does not impact data plane
Baseline is important
15
Troubleshooting CPU from show process cpu
16
CPU higher than
baseline
High iosd use on IOS-XE?
sh pr oc cpu det ai l
pr ocess i osd
No
Reference Document ID: 65591 on
http://www.cisco.com for more
details
High CPU in IOS process or
Cat4k process?
Troubleshoot features related
to the process / open TAC SR
No
Yes
High CPU traffic driven?
(K*CpuMan Review)
show pl at f or mheal t h
ios cat4k
Can the traffic be identified?
show pl at f or mcpu packet st at
No
Yes
Stop / alter traffic source,
open TAC SR if more detail
needed
moni t or sessi on 1 sour ce cpu
OR
debug pl at f or mpacket al l buf f er
show pl at f or mcpu packet buf f er
No Yes
IOS-XE
IOS
Troubleshooting CPU: Narrowing Down Process
swi t ch# show process cpu sort
Cor e 0: CPU ut i l i zat i on f or f i ve seconds: 99%; one mi nut e: 16%; f i ve mi nut es: 7%
PI D Runt i me( ms) I nvoked uSecs 5Sec 1Mi n 5Mi n TTY Pr ocess
8590 3186391 38863326 176 51. 20 42. 52 20. 34 0 i osd

11969 3138594 13447334 23 0. 08 0. 07 0. 05 0 f f m
8448 207801 20750735 10 0. 04 0. 14 0. 27 0 cl i _agent
10684 428406 20858613 20 0. 04 0. 01 0. 01 0 l i censed
11241 3603017 26001138 138 0. 04 0. 04 0. 04 0 cpumemd

swi t ch# show proc cpu detail process iosd sort
PI D T C TI D Runt i me( ms) I nvoked uSecs 5Sec 1Mi n 5Mi n TTY Pr ocess
( %) ( %) ( %)
8590 L 3346604 3886415 176 51. 12 50. 36 32. 75 0 i osd
8590 L 0 8590 3561989 2098956 0 49. 88 49. 04 30. 82 0 i osd
8590 L 1 12314 4076156 1787406 0 1. 24 1. 32 1. 91 0 i osd
8590 L 0 12315 3425 52685 0 0. 00 0. 02 0. 06 0 i osd
24 I 376348 695349 0 77. 00 75. 77 43. 55 0 ARP I nput
85 I 534349 8127080 0 18. 77 18. 77 12. 66 0 Cat 4k Mgmt Hi Pr i
7 I 2083841 1110797 0 1. 11 0. 33 0. 22 0 Check heaps
86 I 744497 5671481 0 1. 11 1. 22 2. 22 0 Cat 4k Mgmt LoPr i
Dual Core
17
IOS-XE processes
Traditional IOS
processes indented
Catalyst-4k Specific
Management Processes
17
Troubleshooting CPU: Packet-Driven CPU
swi t ch# show platform health
%CPU %CPU RunTi meMax Pr i or i t y Aver age %CPU Tot al

Tar get Act ual Tar get Act ual Fg Bg 5Sec Mi n Hour CPU
K5CpuMan Revi ew 30. 00 70. 81 30 17 100 500 91 66 9 19: 17

Swi t ch# show platform cpu packet statistics
Packet s Dr opped by Packet Queue

Queue Tot al 5 sec avg 1 mi n avg 5 mi n avg 1 hour avg
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
I p Opt i on 10715071 118803 71866 15919 0

( conf i g) # monitor session 1 source cpu rx
( conf i g) # monitor session 1 destination interface Gi1/48

K5CpuMan Over Target
Recent flood of packets with IP Options
(not HW routable)
If port is available, get a full capture from CPU
18
Troubleshooting CPU: SPAN not available?
swi t ch# debug platform packet all buffer
pl at f or mpacket debuggi ng i s on

Swi t ch# show platform cpu packet buffered
Tot al Recei ved Packet s Buf f er ed: 1024
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
I ndex 0:
3 days 23: 23: 18: 54927 - RxVl an: 1006, RxPor t : Gi 1/ 1
Pr i or i t y: Nor mal , Tag: No Tag, Event : 11, Fl ags: 0x40, Si ze: 64
Et h: Sr c 00: 00: 0B: 00: 00: 00 Dst 00: 22: 90: E0: D6: FF Type/ Len 0x0800
I p: ver : I pVer si on4 l en: 24 t os: 0 t ot Len: 46 i d: 0 f r agOf f set : 0 t t l : 64 pr ot o: t cp
sr c: 10. 10. 10. 100 dst : 172. 16. 100. 100 hasI pOpt i ons f i r st Fr agment l ast Fr agment
Remai ni ng dat a:
0: 0x0 0x64 0x0 0x64 0x0 0x0 0x0 0x0 0x0 0x0
10: 0x0 0x0 0x50 0x0 0x0 0x0 0x8A 0x37 0x0 0x0
20: 0x0 0x1 0xB5 0x77 0x6A 0x7E

This debug does not require significant CPU overhead
Be sure to use buffer and not log
Newer versions provide human-readable event
Decode on older versions with:
swi t ch# show pl at f or msof t war e cpu event s | i Code| 11
CPU Event Code PE- Q
1 2 I p Opt i on 11 17
19
Troubleshooting CPU: Common Punt Reasons
Common Cause Recommended Solution
Same interface forwarding no ip redirect, or alter topology
ACL logging disable ACL logging, use ACL matching stats or netflow
ACL deny causing switch to send
ICMP unreachable
no ip unreachables
2

Forwarding/Feature exception (out of
TCAM/adj space)
reduce TCAM usage
resize TCAM region (TCAM2/3)
SW-supported feature (i.e.GRE) disable the feature or reduce the amount of traffic
IP packets with TTL<2, IP options disable the offending traffic, regulate source with Control Plane Policing
1
Unexpected control/data traffic Control Plane Policing
1
1.CoPP supported on all legacy supervisors starting 12.2(31)SG, SUP6-E/6L-E /4900M/4948E on 12.2(50)SG , all Sup7E/7L-E/4500X
2.Must be configured on all the L3 interfaces of the switch

20
System Resources
Memory
Leak vs Large Usage
Large usage goes away when condition is no longer present
Leak never decreases
Establish baseline
Collect multiple iterations over recorded interval
Correlate increase with any known activity
21
Troubleshooting Memory: Large Usage
swi t ch# sh authentication session | count Runn
Number of l i nes whi ch mat ch r egexp = 239

swi t ch# sh proc mem detail proc iosd sort | i Hold|Auth Manager
PI D TTY Al l ocat ed Fr eed Hol di ng Get buf s Ret buf s Pr ocess
113 0 870624 125992 837216 0 0 Aut h Manager

swi t ch( conf i g) # int ra gi 1/1 - 48 , gi 2/1 - 48 , gi 3/1 - 48 , gi 4/1 - 48
swi t ch( conf i g- i f - r ange) # shut
swi t ch( conf i g- i f - r ange) # int ra gi 7/1 - 48 , gi 8/1 - 48 , gi 9/1 - 48 , gi 10/1 - 48
swi t ch( conf i g- i f - r ange) # shut
swi t ch( conf i g- i f - r ange) # end

swi t ch# sh authentication session | count Runn
Number of l i nes whi ch mat ch r egexp = 0

swi t ch# sh proc mem detail proc iosd sort | i Auth Manager
147 0 1434488 601760 514088 0 0 Aut h Manager
300Kb not leaked, simply used
22
Troubleshooting Memory
swi t ch# show proc mem sort
Syst emmemor y : 2011604K t ot al , 765920K used, 1245684K f r ee, 85548K ker nel r eser ved
Lowest ( b) : 710864896
PI D Text Dat a St ack Dynami c RSS Tot al Pr ocess
10137 69308 800424 88 236 958000 1017272 i osd
5498 1140 233600 88 2492 40332 309140 f f m

swi t ch# show proc mem detail proc iosd sort
Pr ocessor Pool Tot al : 805306368 Used: 645097888 Fr ee: 160208480
I / O Pool Tot al : 20971520 Used: 361576 Fr ee: 20609944
Cr i t i cal Pool Tot al : 4087852 Used: 40 Fr ee: 4087812
Cr i t i cal Pool Tot al : 106460 Used: 40 Fr ee: 106420
PI D TTY Al l ocat ed Fr eed Hol di ng Get buf s Ret buf s Pr ocess
153 0 1461539184 749742680 307884712 14266252 0 Aut h Manager
0 0 304511544 14111208 272960272 0 0 *I ni t *
185 0 887586464 301222848 31368752 0 0 CDP Pr ot ocol

swi t ch# show proc mem detail proc iosd task 153
Pr ocess I D: 153
Pr ocess Name: Aut h Manager
Tot al Memor y Hel d: 307882352 byt es
Pr ocessor memor y Hol di ng = 307882352 byt es
pc = 0x16FCD45C, si ze = 291258544, count = 4441
pc = 0x16FCF828, si ze = 9378512, count = 143
For Classic IOS, use:
show pr ocess memsor t
show pr ocess mem<pi d>
Auth Manager holding too much
Collect process memory
breakdown for TAC
23
System Resources
TCAM
Check TCAM usage for ACLs, security, L3 routes, PBR, DHCP Snoop, IPSG,
WCCPv2
%C4K_HWACLMAN- 4- ACLHWPROGERR: I nput VOI P_FROM_CE_I Pv6 - har dwar e TCAM l i mi t ,
qos bei ng di sabl ed on r el evant i nt er f ace
%C4K_HWACLMAN- 4- ACLHWPROGERR: I nput Secur i t y: 101 - har dwar e TCAM l i mi t , some
packet pr ocessi ng wi l l be sof t war e swi t ched
C4K_HWACLMAN- 4- ACLHWPROGERRREASON: I nput ( 75/ Nor mal , 1/ Nor mal ) I nval i d Acl -
based Feat ur e - har dwar e TCAM pol i cer s exceeded
24
Monitoring TCAM
swi t ch# show platform hardware acl statistics utilization brief
CAM Ut i l i zat i on St at i st i cs
- - - - - - - - - - - - - - - - - - - - - - - - - -
Used Fr ee Tot al
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
I nput Secur i t y ( 160) 42 ( 2 %) 2006 ( 98 %) 2048
I nput Secur i t y ( 320) 66 ( 3 %) 1982 ( 97 %) 2048
I nput Qos ( 160) 15 ( 0 %) 2033 ( 100%) 2048
I nput Qos ( 320) 14 ( 0 %) 2034 ( 100%) 2048
I nput For war di ng ( 160) 2 ( 0 %) 2046 ( 100%) 2048
I nput Unal l ocat ed ( 160) 0 ( 0 %) 55296 ( 100%) 55296

swi t ch# show platform hardware qos policer utilization
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Pol i cer ut i l i zat i on summar y:
Di r ect i on Assi gned Used Fr ee
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
I nput 2048 ( 12. 5%) 4 ( 0. 1%) 2044 ( 99. 8%)
Out put 2048 ( 12. 5%) 1 ( 0. 0%) 2047 ( 99. 9%)
Fr ee 12288( 75. 0%) 0 ( 0. 0%) 12288( 100. 0%)

Low utilization
25
System Resources
Queue Memory
Reserved queue memory for each linecard, exceeding this eats into global pool
When global pool exhausted, the above message appears
Options:
decrease queue depths on a per port basis
combine classes under the same queue
%C4K_HWPORTMAN- 3- TXQUEALLOCFAI LED: Fai l ed t o al l ocat e t he needed
queue ent r i es f or Gi 6/ 13
26
Monitoring Queue Memory
Entry Sup6-E/6L-E/7L-E Sup7E
Total queue memory 512K 1M
Free Reserve: global pool 100K 100K
CPU, recirc, drop queues 20K 40K
Queue entries per slot
1
x =400K/ nSlots
2
X =860K/nSlots
Queue entries per port on a line card y =x / nPorts
3
y =x/nPorts
Queue entries per class transmit queue z =y/nTxQs
4
z =y/nTxQs
1. In a redundant chassis, two supervisor slots are treated as one
2. nSlots number of Slots
3. nPorts number of Ports in a line card
4. nTxQs number of transmit queues in use
27
Monitoring Queue Memory
swi t ch# show platform software qm
Dr op por t Tx Queue al l ocat i ons ( Si ze: 8184, Base: 0x019008)

Tx Queue al l ocat i ons f or r eci r c por t s ( Si ze: 24576, Base: 0x01D1D0)

CPU Subpor t Tx Queue al l ocat i ons ( Tot al Si ze: 8656)
Super por t Tx Queue space di st r i but i on

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Super por t Sl ot Per cent Base Addr End Addr Ent r i es
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
4 1 10 0x047ED8 0x04C858 18841
5 1 10 0x04C878 0x0511F8 18841
6 1 10 0x051218 0x055B98 18841
7 1 10 0x055BB8 0x05A538 18841
8 0 10 0x0231D0 0x027B50 18841
9 0 10 0x027B70 0x02C4F0 18841
10 0 10 0x02C510 0x030E90 18841
11 0 10 0x030EB0 0x035830 18841
40 1 10 0x05A558 0x05EED8 18841

41 1 10 0x05EEF8 0x063878 18841
42 1 10 0x063898 0x068218 18841
43 1 10 0x068238 0x06CBB8 18841
18841 * 8 QM entries available for
physical slot 2
150728 / 48 =3140 entries/port
>3140 entries will eat into global pool
Drop, Recirc, CPU reservations
28
Troubleshooting System Resources Commands
CLI Purpose
List IOS process CPU % on IOS-XE
show pr oc cpu det ai l pr ocess i osd sor t
Monitor Cat4k platform CPU statistics
show pl at f or mheal t h
show pl at f or mcpu packet st at i st i cs
SPAN packets to/from CPU
moni t or sessi on 1 sour ce cpu
moni t or sessi on 1 dest i nat i on i nt er f ace 
Enable/monitor Cat4k CPU buffer
debug pl at f or mpacket al l buf f er
show pl at f or mcpu packet buf f er ed
Display process memory and buffer
holdings
show pr oc memsor t
show pr ocess mem<pi d>
show buf f er s
Display process memory and buffer
holdings on IOS-XE
show pr oc memdet ai l pr oc i osd sor t
show pr oc memdet ai l pr oc i osd t ask <pi d>
show buf f er s det ai l ed pr ocess i osd
Display Cat4k ACL and policer usage
show pl at f or mhar dwar e acl st at i st i cs ut i l i zat i on br i ef
show pl at f or mhar dwar e qos pol i cer ut i l i zat i on
Display Cat4k queue memory usage show pl at f or msof t war e qm
29
Agenda
Products Overview
Troubleshooting
Method
System Resources
Packet path / loss
VSS
PoE
Netflow
Tools/Tips
Appendix
30
Troubleshooting Packet Loss / Path
Why is any packet sent to port(s), to CPU, or dropped?
Losing packets on the 4k without a clue why?
1. Collect show tech and iterations of the below
2. Step through the platform
1. Identify counters outside of baseline, find an explanation based on counter meaning
2. Identify unexpected platform programming, work upwards
incrementing counters are most useful
Some counters are normal
Baseline data is useful
31
Areas Of Investigation
HW-based
checks

Queue/buffer
failure
PHY, stub, packet
processor, forwarding
engine
show i nt er f aces count er s al l
show pl at f or mhar dwar e i nt er f st at i s
show pl at f or msof t war e i nt er f st at i s
show pl at f or msof t war e i nt er f st ub st at i s
show pl at f or msof t war e i nt er f st ub ct s st at i s al l
show pl at f or mhar dwar e r et r r q
show pl at f or msof t war e dr op- por t
CPU queues CPU controller show pl at f or mcpu packet dr i ver
show pl at f or mcpu packet st at i st i cs
STP L2 lookup show pl at f or mhar dwar e st p vl an <vl an>
L3 entries forwarding lookup show pl at f or mhar dwar e i p r out e [ i pv4| i pv6] net wor k <net > <mask>
show pl at f or mhar dwar e i p r out e [ i pv4| i pv6] host 
ACL input classification,
output classification
show access- l i st <*acl >
show pl at f or mhar dwar e acl i nput ent r i es st at i c
show pl at f or mhar dwar e acl [ i nput | out put ] ent r i es i nt er f ace al l
show pl at f or mhar dwar e acl [ i nput | out put ] ent r i es vl an <vl an> al l
show pl at f or mhar dwar e acl [ i nput | out put ] act i ons <act i on>
L2 entries,
floodsets
L2 lookup show pl at har d mac add <mac>
show pl at har d r et chai n i ndex 
show pl at f or mhar dwar e f l oodset vl an <vl an>
* Ensure HW statistics are enabled (see ACL section)
32
PHY and Stub ASIC
Line Card
Stub ASICs
Front Panel Ports
Supervisor
Layer 1 issues
Malformed frames/packets
Oversubscription
Flow-control
Storm-control

33
Layer 1 Issues
Match speed and duplex
Isolate bad hardware using known good hardware
Specific to end device? Patch/line cord? Front panel port? Linecard?
Exclude patch panel if possible
Peer misbehaving? Sniff wire for malformed frames
swi t ch# show interfaces g5/5 count errors | exclude \ 0\ *0\ *0\ *0
Por t Cr cAl i gn- Er r Dr opped- Bad- Pkt s Col l i si ons Symbol - Er r
Gi 5/ 5 23736730 0 0 0
Por t Under si ze Over si ze Fr agment s J abber s
Por t Si ngl e- Col Mul t i - Col Lat e- Col Excess- Col
Por t Def er r ed- Col Fal se- Car Car r i - Sen Sequence- Er r

See Appendix for Error descriptions
34
Layer 1 Issues
swi t ch# show platform software interface gigabitEthernet 1/1 stub statistics
Xgst ubMan( 0: N- 0) Por t ( 1 ) Rx St at s:
Over r unPacket s : 0
Al i gnment Er r or Packet s : 0
FcsEr r or Packet s : 0
Symbol Er r or Packet s : 0
I nval i dOver si zePacket s : 0
I pv4Hdr ChecksumEr r or Packet s : 0
I pv4Hdr Er r or Packet s : 0
I pv6Hdr Er r or Packet s : 0

swi t ch# show platform software interface gigabitEthernet 1/1 statistics
Super por t 8( Gi 1/ 1- 6) Non- Zer o Sof t war e St at i st i cs
RxSequenceEr r or s : 255
RxSymbol Er r or s : 255
Note: counters may increment during plug / unplug
Platform commands can narrow down stub
ASIC vs packet processor
35
Layer 1 Issues
( conf i g) # logging event link-status global
( conf i g- i f ) # logging event link-status

swi t ch# show platform software interface all | inc downs:|PimPhyport
Gal Gl mPor t ( 0: N/ 21) , Act i ve? : t r ue, Pi mPhypor t Name : Gi 1/ 22, EpmPor t Man Name : EpmPor t Man( 0: N/ 21)
Name( EpmPor t Man( 0: N/ 21) ) , Pi mPhypor t name( Gi 1/ 22 )
#l i nk downs: 41712
swi t ch# show platform software interface gi1/1 mii
0x00 Cont r ol Reg 0x1140

0x01 St at usReg 0x79C9
0x04 Aut oNegAdvReg 0x01E1

0x05 Aut oNegLi nkPar t ner Abi l i t yReg 0x0000
0x06 Aut oNegExpansi onReg 0x0064
0x07 Aut oNegNext PageTr ansmi t Reg 0x2001
0x09 1000BaseTCont r ol Reg 0x0F00

0x0A 1000BaseTSt at usReg 0x0000
Monitor for link flap via syslog
Configurable globally or per-interface
Get total number of flaps since switch boot
Compare with switch uptime
This command should be run twice
Use the second results, decode standard
802.3 registers
36
Oversubscription: stub/supervisor port buffers
completely even traffic flow does not occur in real-world
2:1 1Gbps != (real world) 500 Mbps x 2 ports
2:1 10bps != (real world) 5Gbps x 2 ports

ingress traffic on oversubscribed ports
control on the peer device

egress oversubscription
consider multi-path
max
avg
min
37
Flow control
switch may send pause toward end-device if rx buffer passes high watermark
stub will pause toward supervisor if end-device signals pause
Stub ASICs
Front Panel Ports
Pause
Packet
Processor
Pause
1
2
Drops
3
1. Device sends pause to stub
2. Stub sends pause to packet
processor
3. Packet processor pauses tx-
queue
38
Tx Oversubscription and Flow Control
swi t ch# show interfaces g2/47 counters detail | begin Drops
Por t Tx- Dr ops- Queue- 5 Tx- Dr ops- Queue- 6 Tx- Dr ops- Queue- 7 Tx- Dr ops- Queue- 8
Gi 2/ 47 0 0 0 37748571

swi t ch# show interfaces g2/47 counters detail | begin RxPause
Por t Rx- No- Pkt - Buf f RxPauseFr ames TxPauseFr ames PauseFr amesDr op
Gi 2/ 47 0 130 0 0
Tx oversubscription will result in tx-queue drops
Pause frames from a peer will stop tx-queue processing
Queue 8 is the default queue with
no QoS Configured
39
Rx Oversubscription

swi t ch # show interface gi1/13 | include overrun
0 i nput er r or s, 0 CRC, 0 f r ame, 86432 over r un, 0 i gnor ed

swi t ch# show interface gi1/13 counter all | begin Rx-No
Por t Rx- No- Pkt - Buf f RxPauseFr ames TxPauseFr ames PauseFr amesDr op
Gi 1/ 13 206658 0 0 0

swi t ch# show platform software interface g1/13 stub stat | in Overrun
Over r unPacket s : 206658 ( l ook f or Rx St at s)

RxFifo stub overrun will be seen during Rx oversubscription
packet buffer depletion can also cause Rx-No-Pkt-Buff
40
Packet Processor
Shared Packet
Memory
Line Card
Supervisor
Packet
Processor

Central packet memory exhaustion
Deep transmit queues
Egress oversubscription (example: SPAN)
J umbo frames

%C4K_SWI TCHI NGENGI NEMAN- 4- I PPLLCI NTERRUPTFREELI STBELOWHI PRI ORI TYTHRESHOLD: I PP
LLC f r eel i st Bel owHi Pr i or i t yThr eshol d i nt er r upt Fr eeLi st Count : 2058,
l owest Fr eeCel l Cnt : 0

Has anyone seen a
longer log message?
41
Oversubscription: packet memory exhaustion

Deep buffers and congestion
limited gain (temporary buffering)
switch-global expense (ingress and egress)

1. Deep egress queue fills
2. Packet memory consumed
3. Packet memory unavailable for ingress
Packet Processor

Shared Packet
Memory
Drops
Drops
1
2
3
Full
42
Oversubscription: packet memory exhaustion

Reduced buffers during congestion
limited expense (smaller threshold on given interface)
large gain (no packet memory exhaustion)

Other solutions:
even out packet port distribution
egress policers
Packet Processor

Shared Packet
Memory
Drops
Restricted
43
Packet memory: keeping the FreeList healthy
swi t ch# show platform hardware interface all | include FreeListCount
Fr eeLi st Count : 64268
swi t ch# show platform hardware interface all | include FreeListCount
Fr eeLi st Count : 62100

swi t ch# show interfaces g2/47 counters detail | begin Drops
Por t Tx- Dr ops- Queue- 5 Tx- Dr ops- Queue- 6 Tx- Dr ops- Queue- 7 Tx- Dr ops- Queue- 8
Gi 2/ 47 0 0 0 37748571

( conf i g) # policy-map egress_queue_limit
class class-default
queue-limit 500

( conf i g) # hw-module system max-queue-limit <value>
64K*280 Byte cells in Sup6E, Sup6L-E
128K*256 Byte cells in Sup7E, Sup7L-E
Drop in FreeList will accompany
IPP log message
1. Locate interfaces tail dropping
2. Reduce tx-queue size OR
3. Modify default queue size
44
Forwarding ASIC
Line Card
Supervisor
NFE

CPU

TCAMs

Forwarding
Engine

Stepping through forwarding ASIC stages
Identifying packet destiny
Punt?
Drop?
Forward to where?
Replicate to where?
Working backwards from ASIC counters
45
Packet Loss / Path: Forwarding ASIC
Location Purpose Most Common Platform Troubleshooting Need
IM Input mapping Vlan re-mapping
L2 L2 lookup Layer 2 destination
IC Input classification ACLs (especially static ACL, which evaluate *all* traffic)
For custom ACL, IOS-level CLI typically all that is needed
NF Netflow Platform troubleshooting not commonly required
IP Input policing IOS-level policer counters typically all that is needed
FL Forwarding lookup L3 Multicast replication
OC Output classification IOS-level CLI typically all that is needed
OP Output policing IOS-level policer counters typically all that is needed
OM Output mapping,
replication
Vlan re-mapping
Replication counters useful in very high density scenarios
QM Queueing Tx-queue programming
46
Packet Loss / Path: Input Mapping
IM
L2
IC
NF
IP
FL
OC
OP
OM
QM
Physical / aggregate port mapping
Vlan mapping
swi t ch# show platform mapping ports
I nt er f ace Super por t Subpor t Compact Subpor t I d Por t Set Phypor t Aggpor t Pi mPhypor t
Gi 1/ 1 8 1 20 2 13 8 0
Gi 7/ 48 35 4 210 8 402 Po1( 417) 367

swi t ch# show platform hardware portvlan-map-table interface gigabitEthernet 1/1
Aggpor t ( 8 ) :

- - - - - Por t Vl anDi r ect Tabl e - - - - -
Vl anI d FwdVl anI d Sr cMi ssCt r l TxDr opEn Vl anTagSt r i pEnOnTx
0 0 Sr cMi ssCopyToCpu Fal se Fal se

- - - - - Por t Vl anHashTabl e - - - - -
I ndex Par t i al Aggpor t Vl anI d FwdVl anI d Di r Sr cMi ssCt r l TxDr opEn Vl anTagSt r i pEnOnTx
1568 8 100 200 Rx Sr cMi ssCopyToCpu - Fal se
3188 8 100 200 Tx - Fal se Fal se
All ports on an Etherchannel share an Aggport
Vlan mapping in use
Mapping information used in many platform CLI outputs
47
Packet Loss / Path: Input Mapping / L2 Lookup
IM
L2
IC
NF
IP
FL
OC
OP
OM
QM
Confirm if routing features are enabled on a vlan
swi t ch# show platform hardware rxvlan-map-table vlan 902
Vl an 902:
l 2LookupI d: 902
sr cMi ssI gnor ed: 0
i pv4Uni cast En: 1
i pv4Mul t i cast En: 1
i pv6Uni cast En: 0
i pv6Mul t i cast En: 0
swi t ch# show int vl 902 | i SVI

Har dwar e i s Et her net SVI , addr ess i s 001e. f 73f . f 5bf ( bi a 001e. f 73f . f 5bf )
swi t ch# show mac address-table vlan 902 | i 001e.f73f.f5bf
902 001e. f 73f . f 5bf st at i c i p, i px, assi gned, ot her Swi t ch
swi t ch# show plat hard mac add 001e.f73f.f5bf vlan 902
I ndex Mac Addr ess Vl an Type Si ngl ePor t / Ret I ndex/ Adj I ndex
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
63248 001E. F73F. F5BF 902 Si ngl ePor t Cpu aggpor t ( 4) ND Rout er Addr

IPv4 unicast and multicast routing enabled
SVI MAC present in MAC
table (for unicast routing)
Note: all SVI use the same MAC address on 4k
48
Packet Loss / Path: L2 Lookup
STP state check
SA Learning
swi t ch# show span int gi 7/48 state | i VLAN0002
VLAN0002 f or war di ng

swi t ch# show platform hardware stp vlan 2 | i Gi7/48
Gi 7/ 48 ( 375) For war di ng

swi t ch( conf i g) # no mac address-table learning vlan 100
swi t ch# show platform hardware rxvlan-map-table vlan 100 | i srcMiss
sr cMi ssI gnor ed: 1

swi t ch# show mac add int gi 1/46 | i 902
902 0000. 0500. 0000 dynami c i p, i px, assi gned, ot her Gi gabi t Et her net 1/ 46
902 f f f f . f f f f . f f f f syst emGi 1/ 46, Gi 7/ 48, Swi t ch

swi t ch# show plat hard mac add 0000.0500.0000 | i 0500|Index
27760 0000. 0500. 0000 902 Si ngl ePor t Gi 1/ 46( 53) ND Sr cOr Dst F
IM
L2
IC
NF
IP
FL
OC
OP
OM
QM
no copies will be sent to CPU for MAC source address learning
HW matches SW
49
SA Lookup: port security
swi t ch# show run int gi 3/19
i nt er f ace Gi gabi t Et her net 3/ 19

swi t chpor t access vl an 172
swi t chpor t mode access
swi t chpor t por t - secur i t y
spanni ng- t r ee por t f ast

swi t ch# show platform hardware mac vl 172
Fl ags ar e:
- - - - - - - - - -
D - Dr op
ND - Do not dr op

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2640 0017. 9543. EA7F 172 Si ngl ePor t Gi 3/ 19( 74) ND Sr cOr Dst
49300 0017. 9543. EA7F 172 Si ngl ePor t Wi l dcar dAggpor t D Sr cOr Dst
IM
L2
IC
NF
IP
FL
OC
OP
OM
QM
Traffic sourced from this MAC from any port
other than Gi3/19 will be dropped on vlan 172
50
DA Lookup: private vlan example
swi t ch# show run int gi 3/7
i nt er f ace Gi gabi t Et her net 3/ 7
swi t chpor t pr i vat e- vl an host - associ at i on 100 200
swi t chpor t mode pr i vat e- vl an host
spanni ng- t r ee por t f ast
end

swi t ch# show platform hardware mac add c89c.1d53.612d
Fl ags ar e:
- - - - - - - - - -
D - Dr op
ND - Do not dr op

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
11700 C89C. 1D53. 612D 200 Si ngl ePor t Gi 3/ 7( 62) ND Sr cOr Dst
46352 C89C. 1D53. 612D 100 Si ngl ePor t Gi 3/ 7( 62) ND Sr cOr Dst
51376 C89C. 1D53. 612D 200 Si ngl ePor t Dr op aggpor t ( 8190) D Sr cOr Dst
IM
L2
IC
NF
IP
FL
OC
OP
OM
QM
Traffic toward C89C.1D53.612D on vlan 200
(isolated vlan) will reach the drop port instead
Note: Index order is not lookup order
51
DA Lookup: multicast, broadcast
swi t ch# show mac add multi vlan 902 | i 0100.5e01.0101
902 0100. 5e01. 0101 i gmp Gi 1/ 46, Swi t ch

swi t ch# show plat hard mac add 0100.5e01.0101 | i 0100.5E01.0101|Index
20224 0100. 5E01. 0101 902 Ret 104444

swi t ch# show plat hard ret chain index 104444
Ret I ndex 104444
Ret Wor dI ndex: 522220 Li nk: 1048575( 0xFFFFF) Fi el dsCnt : 1
Suppr essRxVl anBr i dgi ng: t r ue
Vl an: 902 Br i dgeOnl y: N Gi 1/ 46( 53)

Swi t ch# show platform hardware floodset vlan 902
Vl an 902:
Uni cast Fl oodset :
Fl oodToCpu: -
Ret I ndex: 902
Gi 1/ 46( 53) Po1( 417)
IM
L2
IC
NF
IP
FL
OC
OP
OM
QM
unknown unicasts will be flooded to these ports
Multicast traffic to 0100.5e01.0101 replicated
here, unless overridden by L3/ACL
Note since 15.0(2)SG / 3.2.0SG Broadcast is a
per-vlan ffff.ffff.ffff entry instead of a floodset
52
Packet Loss / Path: L2 vs L3 vs ACL
What HW programming will direct the packet?
swi t ch# show platform hardware ip fwdsel summary
L2Val ue == ot her ( por t / RET) ( 0) :
I C
L3 0 1 2 3
0 l 2 i c i c i c
1 l 3 i c i c i c
2 l 3 l 3 i c i c
3 l 3 l 3 l 3 i c
IM
L2
IC
NF
IP
FL
OC
OP
OM
QM
Fwdsel relevant to ACL (ic) only when there is a
redirect action
Example:
L3 entry present, FwdSel=2
ACL redirect entry present, FwdSel=2
Winner =ACL (ic)
L3 Entry
ACL Entry
L2 entry floodset
Depends on fwdsel
> >
53
Packet Loss / Path: Input Classification
SVI and ACL statistics require hardware resources
Not enabled by default
swi t ch# show run
i nt er f ace Vl an902
i p addr ess 92. 92. 92. 1 255. 255. 255. 0
counter
i p access- l i st ext ended deny

deny i p any any
hardware statistics

swi t ch# show platform hardware vlan statistic summary
Regi on Name Fi r st Last Fi r st Last Used Ent r i es Ent r i es
Bl ock Bl ock Ent r y Ent r y Used Fr ee
Si ze 2 Count er s Regi on 0 510 0 0 1 2043
Si ze 4 Count er s Regi on 511 1022 2044 - 0 2048

Vl anSt at sTabl e Pr ogr ammi ng Compl et e: Yes

IM
L2
IC
NF
IP
FL
OC
OP
OM
QM
Enable hardware counters
Ensure resources are available
54
Any ACL-based ingress classification (security, QoS, PBR)
ACL examples: local multicast sources, static ACL, PBR, PACL
swi t ch# show platform hardware acl input entries vlan 902 all
Opcode : 40000 / 40000

I P Sr c : 92. 92. 92. 0 / 255. 255. 255. 0
I P Dst : 224. 0. 0. 0 / 240. 0. 0. 0
Act I dx: 249 St at sI dx: 0 FwdI dx: ( Cpu, Cpu: t r ue, CpuEvent : 1, Por t : 6)

swi t ch# show platform hardware acl input actions 249
I dx: 249
FwdSel : 2
L3Act i on: 2
IM
L2
IC
NF
IP
FL
OC
OP
OM
QM
Installed automatically when PIM is
enabled on the SVI
Matches local sources >TTL=1
Redirects to CPU for S,G setup (if
not overridden by L3 entry)
Compare FwdSel with L3 entries
L3Action: (0 =permit, 1 =drop, 2 =redirect)
55
swi t ch# show platform hardware acl input entries static
CamI ndex Ent r y Type Act i ve Appl y QoS Hi t Count

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2 I gmpToCpu Y N/ A 14237 ( est i mat e)
swi t ch# show platform hardware acl input entries start 2 end 2 all
I P Sr c : 0. 0. 0. 0 / 0. 0. 0. 0
I P Dst : 224. 0. 0. 0 / 240. 0. 0. 0
I P Pr ot ocol : i gmp / I pPr ot ocol Mask
Act I dx: 252 St at sI dx: 0 FwdI dx: ( Cpu, Cpu: t r ue, CpuEvent : 1, Por t : 3)

FwdSel : 3
L2Act i on: 2
IM
L2
IC
NF
IP
FL
OC
OP
OM
QM
Watch for increment
Hit does not mean packet count
IGMP sent to 224/4
will go to CPU
if FwdSel wins over L3
56
swi t ch# show platform hardware acl input entries vlan 901 all
I P Sr c : 1. 1. 1. 1 / 255. 255. 255. 255

I P Dst : 0. 0. 0. 0 / 0. 0. 0. 0
Act I dx: 244 St at sI dx: 0 FwdI dx: ( Adj , Adj : 8)

FwdSel : 2
L3Act i on: 2

swi t ch# show platform hardware ip adjacency entry 8
000008: vl an: 192 por t : Po1 ( 417) si ze: 1 i f aI d: 20
f wdCt r l : 5 cpucode: 3 si f act 4: FwdToCpu si f act 6: FwdToCpu
sa: 00: 1E: F7: 3F: F5: BF da: 00: 0C: 29: 6D: 1A: ED r wFmt : Uni cast
packet s: 0 byt es: 0
IM
L2
IC
NF
IP
FL
OC
OP
OM
QM
Packets sourced from 1.1.1.1/32
will be redirected to adjacency 8 (Po1)
If FwdSel wins over L3
Note: PBR ACLs are removed if
adjacency becomes unavailable
57
Note: packets classified as non-IP, IPv4, IPv6 (cannot MAC ACL on an IP packet)
swi t ch# show ip access deny
Ext ended I P access l i st deny
10 deny i p any any ( 1056 mat ches)
swi t ch# show ip int gi 1/2
I nbound access l i st i s deny
swi t ch# show plat hard acl inp entr int gi 1/2 all
I P Sr c : 0. 0. 0. 0 / 0. 0. 0. 0
I P Dst : 0. 0. 0. 0 / 0. 0. 0. 0
I P Pr ot ocol : I pPr ot ocol Nul l / I pPr ot ocol Nul l
Act I dx: 254 St at sI dx: 0 FwdI dx: ( None, r ep: 0)

swi t ch# show plat hard acl inp act 254
FwdSel : 0
L2Act i on: 1
IM
L2
IC
NF
IP
FL
OC
OP
OM
QM
All IPv4 traffic will be dropped
Fwdsel doesnt matter
58
Packet Loss / Path: Input Classification / Policing
Order of operations
f l ow r ecor d mi cr of l ow
mat ch i pv4 sour ce addr ess
cl ass- map mat ch- al l mi cr of l ow
mat ch f l ow r ecor d mi cr of l ow
pol i cy- map i ngr ess
cl ass voi ce- si gnal l i ng
set dscp cs3
pol i ce ci r 32000 bc 8000
conf or m- act i on t r ansmi t
exceed- act i on set - dscp- t r ansmi t cs1
exceed- act i on set - cos- t r ansmi t 1
cl ass mi cr of l ow
pol i ce ci r 100000
exceed- act i on dr op
cl ass cl ass- def aul t
set dscp def aul t
set cos 0
IM
L2
IC
NF
IP
FL
OC
OP
OM
QM
Unconditional Marking
Microflow policing
Flexible Netflow
Class-map matching FNF
Policer
Normal policer
Conditional Marking
Classification
Ingress
Classification
Ingress Policing
Ingress Marking
Unconditional
Ingress Marking
Conditional
Forwarding
59
Packet Loss / Path: Input Classification / Policing
Monitoring ingress Qos
swi t ch# show policy-map interface gigabitEthernet 1/46
Gi gabi t Et her net 1/ 46

Ser vi ce- pol i cy i nput : i ngr ess

Cl ass- map: voi ce- si gnal l i ng ( mat ch- al l )
28283457437 packet s
Mat ch: dscp ef ( 46)
QoS Set
dscp cs3
pol i ce:
ci r 32000 bps, bc 8000 byt es
conf or med 76128704 byt es; act i ons:
t r ansmi t
exceeded 1810581188160 byt es; act i ons:
set - dscp- t r ansmi t cs1
set - cos- t r ansmi t 1
conf or med 32000 bps, exceed 761238000 bps
IM
L2
IC
NF
IP
FL
OC
OP
OM
QM

Class-map stats are shared across interfaces with the
same policy map
Ensure counters increment
Classification displays using the packet counts
Policing displays using bytes
60
Packet Loss / Path: Forwarding Lookup
L3 unicast destination lookups, multicast (*,G) / (S,G) lookups, urpf lookups
swi t ch# show ip route 192.168.200.200
Rout i ng ent r y f or 192. 168. 200. 0/ 24
Known vi a " st at i c" , di st ance 1, met r i c 0
Rout i ng Descr i pt or Bl ocks:
* 192. 168. 100. 100
Rout e met r i c i s 0, t r af f i c shar e count i s 1
swi t ch# show ip arp | i 192.168.100.100
I nt er net 192. 168. 100. 100 0 000c. 296d. 1aed ARPA Vl an192
swi t ch# show mac address dynamic | i 000c.296d.1aed
192 000c. 296d. 1aed dynami c i p, i px, assi gned, ot her Por t - channel 1
swi t ch# show platform hardware ip route ipv4 network 192.168.200.0 255.255.255.0
Bl ock: 0 En: t r ue Ent r yMap: LSB Wi dt h: 80- Bi t Type: Dst
000022: v4 192. 168. 200. 0/ 24 - - > vr f : Gl obal Rout i ng Tabl e ( 0)

adj St at s: t r ue f wdSel : 2 mr pf : 0 ( None) f wdI dx: 0 t s: 0
adj I ndex: 8 vl an: 192 por t : Po1 ( 417)
f wdCt r l : 5 cpucode: 3 si f act 4: FwdToCpu si f act 6: FwdToCpu
sa: 00: 1E: F7: 3F: F5: BF da: 00: 0C: 29: 6D: 1A: ED
IM
L2
IC
NF
IP
FL
OC
OP
OM
QM
Remember: unicast traffic wont be destination-routed unless:
routing is enabled on the vlan
traffic is sent to L3 MAC
FwdSel of route wins over ACL
62
In general: (S,G) >vlan multicast source ACL >(*,G)

swi t ch# show ip mroute 239.1.1.1 91.91.91.100
( 91. 91. 91. 100, 239. 1. 1. 1) , 00: 08: 11/ 00: 01: 32, f l ags: J T
I ncomi ng i nt er f ace: Vl an901, RPF nbr 0. 0. 0. 0
Out goi ng i nt er f ace l i st :
Vl an902, For war d/ Spar se, 00: 07: 49/ 00: 02: 53

swi t ch# show platform hardware ip route ipv4 host 239.1.1.1
008194: v4 91. 91. 91. 100/ 32 239. 1. 1. 1/ 32 - - > vr f : Gl obal Rout i ng Tabl e ( 0)
adj St at s: t r ue f wdSel : 3 mr pf : 901 ( FwdToCpu) f wdI dx: 0 t s: 0
r et I ndex: 49150 r et Ts: 0
Vl an: 901 Br i dgeOnl y: Y Gi 1/ 46( 53)
Vl an: 901 Br i dgeOnl y: Y Gi 7/ 1( 328)
Vl an: 901 Br i dgeOnl y: Y Po1( 417)
Vl an: 902 Br i dgeOnl y: N Gi 1/ 46( 53)
IM
L2
IC
NF
IP
FL
OC
OP
OM
QM
BridgeOnly =Y, packet will be bridged (to Gi1/46 vlan 901)
BridgeOnly =N, packet will be routed (to Gi1/46 vlan 902)
Packets matching the (S,G) NOT ingressing
mrpf vlan will fail rpf check, punt to CPU
63
Quiz scenario:
Switch configured for multicast routing, sparse mode
No RP address is configured
A local multicast source starts
IM
L2
IC
NF
IP
FL
OC
OP
OM
QM
Why does the new
source stream to the
CPU?
Answer:
Vlan local source ACL punts traffic to the CPU
No S,G is ever created to override the ACL (via fwdsel)
64

swi t ch# show run int vl 901
i nt er f ace Vl an901
i p addr ess 91. 91. 91. 1 255. 255. 255. 0
i p ver i f y uni cast sour ce r eachabl e- vi a r x al l ow- def aul t

swi t ch# show platform hardware ip route ipv4 network 91.91.91.0 255.255.255.0
Bl ock: 3 En: t r ue Ent r yMap: LSB Wi dt h: 80- Bi t Type: Sr c
012333: v4 91. 91. 91. 0/ 24 * - - > vr f : Gl obal Rout i ng Tabl e ( 0)

def aul t Rout e: f al se r pf Vl an: 901 ( Dr op) t s: 0

IM
L2
IC
NF
IP
FL
OC
OP
OM
QM
Routed traffic sourced from 91.91.91.0/24
Where RPF fails (ie doesnt ingress vlan 901)
Will be dropped
65
Packet Loss / Path: Output Classification, Policing, Mapping
ACL-based output classification: Security, qos
ACL and policer CLI the same (change input ->output)
Mapping behavior shown in ingress mapping CLI
STP state is not checked in HW
entries/floodsets simply dont include ports in blocked state
Packets for replication enqueued into replication queue

IM
L2
IC
NF
IP
FL
OC
OP
OM
QM
swi t ch# show platform hardware ret rrq
ReasonQueue i s not empt y

ReasonHead: 0xDEB
ReasonTai l : 0xDD7
Dat aQueue i s not empt y
Dat aHead: 0xDE2
Dat aTai l : 0xDD7
Pr ef ul l dr ops: 171477
Over Thr eshol d dr ops: 0
Control, lookup queues are in use
Drops have occurred due to reaching first drop threshold
66
Packet Loss / Path: Output Classification / Policing
Order of operations
pol i cy- map egr ess
cl ass voi ce
set dscp ef
set cos 5
pr i or i t y
pol i ce ci r per cent 33
cl ass voi ce- cont r ol
set dscp af 31
set cos 3
bandwi dt h r emai ni ng per cent 5
dbl
IM
L2
IC
NF
IP
FL
OC
OP
OM
QM
Marking
Queuing
Policing

Note: dbl, shape, bandwidth, queue-limit and priority
commands are all queuing commands
MQC for port-channels:

Policy with queuing actions only physical ports
Policy with non-queuing actions only port channel
Output
Classification
Output Policing
Output Marking
Unconditional
Output Marking
Conditional
Queuing
Classification
67
Packet Loss / Path: Output Classification / Policing
Monitoring egress Qos
swi t ch# show pol i cy- map i nt g1/ 36 out put
Gi gabi t Et her net 1/ 36

Ser vi ce- pol i cy out put : Aut oQos- VoI P- Out put - Pol i cy

Cl ass- map: Aut oQos- VoI P- Bear er - QosGr oup ( mat ch- al l )
625530530 packet s
Mat ch: qosgr oup 46
QoS Set
i p dscp ef
cos 5
pr i or i t y queue:
Tr ansmi t : 32344068480 Byt es, Queue Ful l Dr ops: 0 Packet s
pol i ce:
ci r 33 %
ci r 330000000 bps, bc 10312500 byt es
conf or med Packet count - n/ a, 32335870400 byt es; act i ons:
t r ansmi t
exceeded Packet count - n/ a, 7813435520 byt es; act i ons:
dr op
conf or med 325185000 bps, exceed 97368000 bps
IM
L2
IC
NF
IP
FL
OC
OP
OM
QM
Class-map stats are shared across interfaces with the
same policy map
Ensure counters increment
Classification display using the packet counts
Policing display using bytes
Queue full drops are in packets
68
Packet Loss / Path: Output Queuing
DBL processing (if packet is not scheduled for drop)
Descriptor enqueued in queue memory
swi t ch# show platform hardware interface gigabitEthernet 1/1 tx-queue
Phypor t TxQ Head Tai l Pr e Empt y Num BaseAddr Si ze Shape- Ok

Empt y Packet s TxQ Subpor t
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Gi 1/ 1 0 0x0000 0x0000 Tr ue 0 0x20D10 16 Tr ue Tr ue
Gi 1/ 1 1 0x0000 0x0000 Tr ue 0 0x00000 0 Tr ue Tr ue
Gi 1/ 1 7 0x0000 0x0000 Tr ue 0 0x20D20 3152 Tr ue Tr ue
IM
L2
IC
NF
IP
FL
OC
OP
OM
QM
Reminder: SPAN copies are probably sent
to different port / queues
Default queues configured
Currently empty
69
pol i cy- map egr ess_queuei ng
cl ass dscp32- 48
pol i ce ci r 990000
exceed- act i on dr op
pr i or i t y
cl ass dscp0- 15
bandwi dt h 250000
queue- l i mi t 400
cl ass dscp16- 31
bandwi dt h 250000
queue- l i mi t 512

swi t ch# show platform hardware interface g2/48 tx-queue
Phypor t TxQ Head Tai l Pr e Empt y Num BaseAddr Si ze Shape- Ok

Empt y Packet s TxQ Subpor t
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Gi 2/ 48 0 0x0000 0x0000 Tr ue 0 0x5ECE8 352 Tr ue Fal se
Gi 2/ 48 1 0x0000 0x0000 Tr ue 0 0x00000 0 Tr ue Fal se
Gi 2/ 48 5 0x0000 0x0000 Tr ue 0 0x5E958 512 Tr ue Fal se
Gi 2/ 48 6 0x0000 0x0000 Tr ue 0 0x5EB58 400 Tr ue Fal se
Gi 2/ 48 7 0x008A 0x0088 Fal se 1421 0x5EE48 1520 Tr ue Fal se
Packet Loss / Path: Output Queuing
Tx Q Class
0 dscp32-48
5 dscp16-31
6 dscp0-15
7 dscp49-63, class-default
Low priority queues can be
starved, policer recommended
Last queue is default queue
In this example, it is non-empty
First and last appear where expected, middle reversed
70
Packet Loss / Path: ASIC Drop Categories
Common Drop Event Reason Typical Description
BridgeToRxPortDrop received in a vlan with no other ports, replicated to a floodset/entry where ingress port
was a member
DblDrop packets dropped by DBL (including DBL on CPU ports)
InpL2AclDrop, InpL3AclDrop,
OutL2AclDrop, OutL3AclDrop
packets denied by ACL
rplErrDrop broadcast/multicast packets dropped while being replicated, many normal reasons to
increment, including: rpf failure, floodset containing drop port, packets replicated to the
CPU but also bridged to a floodset/entry containing the CPU
SptDrop spanning-tree drop; packets dropped because a port is not in a forwarding state
SrcHitDrop dropped at source learning stage; example: static MAC drop entry
TxQueFullDrop a tx port is oversubscribed
show pl at f or msof t war e dr op- por t shows global ASIC drop events (not per interface)
these counters are frequently expected
baseline and/or high packet rate very useful
71
Packet Loss / Path: CPU Queues
72
swi t ch# show plat cpu pack driv
For er unner Packet Engi ne 1. 83 ( 0)

Recei ve Queues: r ecei ved packet s summar y
Qu Capac Guar a Cur Po Unpr o Accum Kept Bper P Packet s
2 2512 112 610 0 2 2 73 610
58 512 256 37 12 5 511 216 591103

Recei ve Queues: dr opped packet s summar y
Qu Tot al Packet s Dr op No Cel l Dr op Over r un Dr op Under r un
58 591103 43623295103 0 0

Tr ansmi t Queues
Qu PosAdd Pendng Packet s Byt es
0 595 0 8633668179 663318795241
1 863 0 5315423 363150782
However, combine high Kept with:
CurPo does not increment
Drop No Cell does increment
queue 58 is stuck!
High Kept indicates high rate of traffic
Incrementing Drop No Cell indicates
queue oversubscription
Check for transient flooding / loss versus stuck queue
Decode queue meaning with show pl at f or msof t war e cpu event s
Agenda
Products Overview
Troubleshooting
Method
System Resources
Packet path / loss
VSS
PoE
Netflow
Tools/Tips
Appendix
73
Troubleshooting VSS
Core
Distribution
VSS
member 1
VSS
member 2
Access
VSL
Core Switch 1 Core Switch 2
Access Switch 1 Access Switch 3 Access Switch 2
Differences
VSL Health
Packet Path
74
Troubleshooting VSS: Tips and Differences
Available on Sup7E/4500X (ipbase or better), Sup7L-E (entservices or better)
No quad-sup SSO, but you can use in-chassis standby (ICS) uplinks
Configure VSS before installing ICS
ICS must remain in rommon
Split-brain detection uses ePAGP
MEC policers are applied independently (eg 100Mbps =100 @ active, 100 @ standby)
No qos groups
Not currently supported: smart Install, linecards prior to 46**, custom VSL qos
75
swi t ch# show redundancy | i Current
Cur r ent Pr ocessor I nf or mat i on :
Cur r ent Sof t war e st at e = ACTI VE
Cur r ent Sof t war e st at e = STANDBY HOT

swi t ch# show switch virtual
Execut i ng t he command on VSS member swi t ch r ol e = VSS Act i ve, i d = 1

Swi t ch mode : Vi r t ual Swi t ch
Vi r t ual swi t ch domai n number : 100
Local swi t ch number : 1
Local swi t ch oper at i onal r ol e: Vi r t ual Swi t ch Act i ve
Peer swi t ch number : 2
Peer swi t ch oper at i onal r ol e : Vi r t ual Swi t ch St andby

Execut i ng t he command on VSS member swi t ch r ol e = VSS St andby, i d = 2

Swi t ch mode : Vi r t ual Swi t ch
Vi r t ual swi t ch domai n number : 100
Local swi t ch number : 2
Local swi t ch oper at i onal r ol e: Vi r t ual Swi t ch St andby
Peer swi t ch number : 1
Peer swi t ch oper at i onal r ol e : Vi r t ual Swi t ch Act i ve
Troubleshooting VSS: VSL Health
Chassis SSO is established
VSS is functioning
76
Troubleshooting VSS: VSL Health
swi t ch# show switch virtual link port-channel | i Po
Gr oup Por t - channel Pr ot ocol Por t s
10 Po10( SU) - Te1/ 3/ 1( P) Te1/ 3/ 2( P)
20 Po20( SU) - Te2/ 3/ 1( P) Te2/ 3/ 2( P)
Gr oup Por t - channel Pr ot ocol Por t s
10 Po10( SU) - Te1/ 3/ 1( P) Te1/ 3/ 2( P)
20 Po20( SU) - Te2/ 3/ 1( P) Te2/ 3/ 2( P)
swi t ch# show policy-map int te1/3/2 | i Class|drops
Cl ass- map: VSL- MGMT- PACKETS ( mat ch- any)
( queue dept h/ t ot al dr ops) 0/ 0
Cl ass- map: VSL- L2- CONTROL- PACKETS ( mat ch- any)
Cl ass- map: VSL- L3- CONTROL- PACKETS ( mat ch- any)
Cl ass- map: VSL- VOI CE- VI DEO- TRAFFI C ( mat ch- any)
Cl ass- map: VSL- SI GNALI NG- NETWORK- MGMT ( mat ch- any)
Cl ass- map: VSL- MULTI MEDI A- TRAFFI C ( mat ch- any)
Cl ass- map: VSL- DATA- PACKETS ( mat ch- any)
Cl ass- map: cl ass- def aul t ( mat ch- any)
VSL members bundled
Watch for non-zero queue depth or incrementing drops on control queues
Drops on non-control queues? Increase VSL links/speed
77
Troubleshooting VSS: Packet Path
swi t ch# show platform hardware floodset vlan 97
Execut i ng t he command on VSS member swi t ch r ol e = VSS

Act i ve, i d = 1

Vl an 97:
Fl oodToCpu: -
Ret I ndex: 97
Gi 1/ 5/ 69( 236) Al t er nat e VSL aggpor t ( 1528)
I pv4 Mul t i cast Fl oodset :

Fl oodToCpu: N
Ret I ndex: 16481
Gi 1/ 5/ 69( 236) Po10( 842)

Execut i ng t he command on VSS member swi t ch r ol e = VSS
St andby, i d = 2

Vl an 97:
Fl oodToCpu: -
Ret I ndex: 97
Al t er nat e VSL aggpor t ( 1528) Gi 2/ 1/ 1( 420)
Gi 2/ 7/ 38( 777)
I pv4 Mul t i cast Fl oodset :

Fl oodToCpu: N
Ret I ndex: 16481
Gi 2/ 1/ 1( 420) Gi 2/ 7/ 38( 777) Po20( 1108)
VSS virtual data path visible in platform programming
Reflected in all packet path programming
If traffic needs to cross chassis, VSL aggport,
VSL Po, or CPU must be used
78
Agenda
Products Overview
Troubleshooting
Method
System Resources
Packet path / loss
VSS
PoE
Netflow
Tools/Tips
Appendix
79
Troubleshooting PoE: Power Supply and Linecards
swi t ch# show environment status
<sni p>
Super vi sor Led Col or : Gr een

Modul e 1 St at us Led Col or : Gr een
Modul e 2 St at us Led Col or : Gr een PoE Led Col or : Gr een

swi t ch# show power detail
Power Fan I nl i ne
Suppl y Model No Type St at us Sensor St at us
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
PS1 PWR- C45- 4200ACV AC 4200W good good good
PS1- 1 110V good
PS1- 2 110V good
PS2
Wat t s Used of Syst emPower ( 12V)
Mod Model budget ed i nst ant aneous peak out of r eset i n r eset
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 WS- X4648- RJ 45V- E 92 - - - - 92 10
2 WS- X4548- GB- RJ 45V 60 - - - - 60 25

PoE is
operational on
the line card
If not good,
check power
supply LEDs
Linecards are
fully powered
80
Troubleshooting PoE: Analyze Power Budget
swi t ch# show power detail
Power Summar y Maxi mum
( i n Wat t s) Used Avai l abl e
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Syst emPower ( 12V) 847 1360
I nl i ne Power ( - 50V) 6 1580
Backpl ane Power ( 3. 3V) 40 40
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Tot al 893 ( not t o exceed Tot al Maxi mumAvai l abl e = 2100)
I nl i ne Power Admi n I nl i ne Power Oper
Mod Model PS Devi ce PS Devi ce Ef f i ci ency
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 WS- X4648- RJ 45V- E 7 6 9 8 93
2 WS- X4548- GB- RJ 45V 0 0 17 15 89
Tot al 7 6 26 23
PoE Allocated
Inline power available. If not, this log would be seen:
%I LPOWER- 5- I LPOWER_POWER_DENY: I nt er f ace
: i nl i ne power deni ed
Switch will allocate highest power level requested by the phone
Catalyst 4500 power allocation rules:
Power line cards before IP phones
Prefer static over auto power
Cisco Power Calculator:
http://tools.cisco.com/cpc/launch.jsp
81
Troubleshooting PoE: Linecard Status
swi t ch# show module
Chassi s Type : WS- C4510R- E
Power consumed by backpl ane : 40 Wat t s
Mod Por t s Car d Type Model Ser i al No.
- - - +- - - - - +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +- - - - - - - - - - - - - - - - - - +- - - - - - - - - - -
1 48 10/ 100/ 1000BaseT POE E Ser i es WS- X4648- RJ 45V- E J AE1329EAVL
2 48 10/ 100/ 1000BaseT ( RJ 45) V, Ci sco/ I EEE WS- X4548- GB- RJ 45V J AE10244L7P
4 18 10GE ( X2) , 1000BaseX ( SFP) WS- X4606- X2- E J AE12021FMP
5 6 Sup 6- E 10GE ( X2) , 1000BaseX ( SFP) WS- X45- SUP6- E J AE1223KL3G
6 6 Sup 6- E 10GE ( X2) , 1000BaseX ( SFP) WS- X45- SUP6- E J AE12460E61
M MAC addr esses Hw Fw Sw St at us
- - +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +- - - +- - - - - - - - - - - - +- - - - - - - - - - - - - - - - +- - - - - - - - -
1 0024. 1446. 2d93 t o 0024. 1446. 2dc2 1. 0 Ok
2 0018. 1958. cf 70 t o 0018. 1958. cf 9f 3. 3 Ok
4 001d. 4573. 0ada t o 001d. 4573. 0aeb 1. 0 Ok
5 0022. 90e0. d6c0 t o 0022. 90e0. d6c5 1. 1 12. 2( 44r ) SG 12. 2( 53) SG1 Ok
6 0022. 90e0. d6c6 t o 0022. 90e0. d6cb 1. 2 12. 2( 44r ) SG 12. 2( 53) SG1 Ok

If not Ok, try resetting after executing all
troubleshooting steps:
hw- modul e modul e <modul e> r eset
Other status includes: Faulty,
Authfail, Offline, PwrOver, PwrMax,
PwrDeny. See Appendix for details.
82
Troubleshooting PoE: Devices Drawing Too Much
( conf i g- i f ) # power inline police

swi t ch# %I NLI NEPOWEROVERDRAWN: I nl i ne power ed devi ce connect ed on por t Gi 2/ 2 exceeded i t s pol i ced
t hr eshol d.
ERR_DI SABLE: i nl i ne- power er r or det ect ed on Gi 2/ 2, put t i ng Gi 2/ 2 i n er r - di sabl e st at e

swi t ch# show power inline police g2/2
Avai l abl e: 1580( w) Used: 77( w) Remai ni ng: 1503( w)

I nt er f ace Admi n Oper Admi n Oper Cut of f Oper
St at e St at e Pol i ce Pol i ce Power Power
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Gi 2/ 2 aut o er r di sabl e er r di sabl e over dr awn 0. 0 0. 0
( conf i g- i f ) # power inline static max 20000
Policing available from 12.2(50SG)
For phones that rarely draw more than allowed, configure static power
83
Troubleshooting PoE: CDP / LLDP Negotiation
( conf i g) # lldp run
( conf i g) # int gi 3/1
( conf i g- i f ) # lldp tlv-select power-management
Cat 4K Feature Release
LLDP 802.1ab 12.2(44)SG
LLDP 802.3at PoE+TLV, LLDP-MED 12.2(54)SG
Power Negotiation can occur via CDP, LLDP 802.3at or LLDP-MED
Switch "locks" to first protocol packet (CDP or LLDP) that has the power negotiation TLV
LLDP 802.3at power negotiation TLV overrides the LLDP-MED power negotiation TLV
Recommend - disable all but the desired power negotiation protocols on the switch interface & peer
84
Troubleshooting PoE: Verify Data, Collect Debugs
Change connections
Record results of different line card, port, cable, end device
Is this a PoE issue or a PoE and data issue?
Disconnect phone, and connect non-PoE device
Configure power inline never on the port
Verify the link comes up
Re-enable power
Collect additional debugs

swi t ch# show platform chassis module <id>
swi t ch# debug interface g1/48
Condi t i on 1 set
swi t ch# debug ilpower powerman
di sconnect PD, connect PD, col l ect debugs)
swi t ch# undebug all
Al l possi bl e debuggi ng has been t ur ned of f
swi t ch# undebug interface g1/48
Power device (PD)/phone not powering up at all?
Confirm the device is IEEE compliant, check with
vendor
Validate with 3
rd
party PD testers
Device capacitance or impedance as per IEEE?
When PoE is enabled on a port, auto MDIX is disabled.
Please make sure you use the correct cable type. See
the note in the Catalys4500 configuration guide.
85
Troubleshooting PoE: Analyze Power Allocation
Line Card PoE per Line Card PoE per Port
WS-X4748-UPOE+E 1440 60W
WS-X4748-RJ 45V+E 1440 30W
WS-X4648-RJ 45V+E 750 W 30 W
WS-X4548-RJ 45V+ 1050 W 30 W
WS-X4648-RJ 45V-E 750 W 20 W
WS-X4548-GB-RJ 45V 750 W 15.4 W
WS-X4524-GB-RJ 45V 750 W 15.4 W
WS-X4248-RJ 45V 750 W 15.4 W
WS-X4248-RJ 21V 750 W 15.4 W
WS-X4224-RJ 45V 750 W 15.4 W
WS-X4148-RJ 45V 750 W 7 W
WS-X4148-RJ 21V 750 W 7 W
Does the PoE line card support enough
power per port?

Does the PoE line card support enough
power? ( slots 3-10 pair limit in 4510)
Catalyst 4500 Line Cards Data Sheet:
http://www.cisco.com/en/US/prod/collateral/modules/ps2710/ps5494/product_data_sheet0900aecd802109ea_ps4324_Products_Data_Sheet.html

IP Phone Data Sheets:
http://www.cisco.com/en/US/products/hw/phones/ps379/products_data_sheets_list.html.
86
Troubleshooting PoE Commands
Troubleshooting Steps Commands
Check Link debounce settings show interfaces debounce
Check number of debounce events show platform software interfaces mii | inc Debounce
Check Digital Optical Monitoring Data show interface <>transceiver detail
Verify PoE line card is online show module
Verify inline power available and operational show power detail
Verify the inline power status of the port show power inline <interface>[detail]
Verify PoE line card supports enough power per port, per slot Appendix table, line card datasheets
Verify phone is not drawing more power than it should show power inline police <interface>
Verify power negotiation is successful debug interface <interface>
debug ilpower powerman
undebug all
undebug interface <interface>
Gather various module specific debugs show platform chassis module <id>
87
Agenda
Products Overview
Troubleshooting
Method
System Resources
Packet path / loss
VSS
PoE
Netflow
Tools/Tips
Appendix
88
Troubleshooting Flexible Netflow
Overview
Flexible NetFlow (FnF) available on switch, Sup7L-E and 4500X-32
Original netflow src/dst IP, src/dst L4 port, protocol, TOS, and input interface
Flexible netflow user defined fields (supports L2, IPv4, IPv6)
Support both v9 (flexible) and v5 (fixed tuple) export formats
Uses
Troubleshooting profile for suspected patterns and port
Network security monitor and record network meta-data, spot new patterns
Usage monitoring and billing
89
Troubleshooting FNF Export
Flow stats not received at collector
UDP export only, check for packet loss along path to collector
Issue can be with the collector as well
Confirm NetFlow export version matches the collector
Note mandatory fields are required for v5 export
( conf i g) # flow exporter flowexporter1
( conf i g- f l owexpor t er ) # destination 10.10.22.22
( conf i g- f l owexpor t er ) # export-protocol netflow-v5
( conf i g- vl an- conf i g) # ip flow monitor flowmonitor1 input
War ni ng: Expor t er f l owexpor t er 1 coul d not be act i vat ed because t he f ol l owi ng f i el ds ar e mandat or y:
i pv4 sour ce addr ess
i pv4 dest i nat i on addr ess
t r anspor t sour ce- por t
t r anspor t dest i nat i on- por t
i pv4 pr ot ocol
90
Troubleshooting FNF Export
Flows stats may be lost if there are more flows than permitted in the monitor cache
Constant cache aging on flow monitors can also drive CPU higher
swi t ch# show flow monitor ipv4fm cache
Cache t ype: Nor mal
Cache si ze: 4096
Cur r ent ent r i es: 3891
Hi gh Wat er mar k: 4096

Fl ows added: 12288
Fl ows aged: 8397
- Act i ve t i meout ( 1800 secs) 0
- I nact i ve t i meout ( 15 secs) 0
- Event aged 0
- Wat er mar k aged 599
- Emer gency aged 7798
( conf i g- i f ) # no ip flow monitor ipv4fm input
( conf i g- i f ) # exit
( conf i g) # flow monitor ipv4fm
( conf i g) # cache entries 64000
( conf i g) # int gi 1/46
( conf i g- i f ) # ip flow monitor ipv4fm input

swi t ch# show flow monitor ipv4fm cache
Cache t ype: Nor mal
Cache si ze: 64000
Cur r ent ent r i es: 32768
Hi gh Wat er mar k: 32768

Fl ows added: 32768
- Emer gency aged 0

Tune cache size to match flow flux
91
Troubleshooting Monitoring FNF Collisions
Limit cache entries
If cache limit is already reached and
hash table is full, scope of
monitoring will need to be adjusted
swi t ch# show platform hardware flow table utilization
Bucket s w/ X Bucket Count Used Ent r y Count

Used Ent r i es ( %of Bucket s) ( %of Ent r i es)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
0 0 ( 0. 0) 0 ( 0. 0)
1 0 ( 0. 0) 0 ( 0. 0)
2 0 ( 0. 0) 0 ( 0. 0)
3 0 ( 0. 0) 0 ( 0. 0)
14 0 ( 0. 0) 0 ( 0. 0)
15 1 ( 0. 0) 15 ( 0. 0)
16 8191 ( 99. 9) 131056 ( 99. 9)
Tot al Used 8192 ( 100. 0) 131071 ( 99. 9)
Tot al Fr ee N/ A 1 ( 0. 0)

Unaccount ed packet s:
User conf i gur ed f l ow moni t or cache l i mi t r eached:
4419746531
I Pv6 ent r y t abl e f ul l : 0
Hash Col l osi ons: 176000251

Flow Hash Table Buckets 8K
Entries per bucket 16
Total hash table entries 128K
Approx. total usable space 108K
%C4K_HWFLOWMAN-5-
FLOWUNACCOUNTEDPACKETS: Flow stats for
46444030 packets are not accounted due to hardware
hash collisions or full hardware flow table
All 16-entry buckets are
full =constant collisions
92
Agenda
Products Overview
Troubleshooting
Method
Packet path / loss
VSS
PoE
System Resources
Netflow
Tools/Tips
Appendix
93
Tools: Wireshark
Wireshark Best Practices
Do not display directly to console without a buffer, file or a duration limit
Write to PCAP file on storage, display on switch or using laptop Wireshark GUI
Only the core filter is implemented in hardware as ACLs. Use a restricted filter to avoid high CPU
Available on Sup7E, Sup7L-E, 4500X
Onboard full packet capture, filter, decode / display
Up to 8 instances supported
94
Tools: Wireshark
Forwarding
Engine

IOS-XE
Ring Buffer
Console
File
Core Filter
Display
Filter
Display
Filter
Capture
Filter
swi t ch# monitor capture mycap int gi 1/46 in match ipv4 protocol tcp 10.1.1.1/32 any file location
bootflash:mycap.pcap limit duration 3

swi t ch# monitor capture mycap start
*Apr 15 17: 56: 24. 291: %BUFCAP- 6- ENABLE: Capt ur e Poi nt mycap enabl ed.
*Apr 15 17: 56: 27. 720: %BUFCAP- 6- DI SABLE_ASYNC: Capt ur e Poi nt mycap di sabl ed. Reason : Wi r eshar k sessi on
ended

swi t ch# show monitor capture file bootflash:mycap.pcap display-filter "ip.ttl == 100
1 0. 000000 10. 1. 1. 1 - > 91. 91. 91. 100 TCP [ TCP Zer oWi ndow] 0 > 0 [ <None>] Seq=1 Wi n=0 Len=2
95
Tools: Wireshark
Troubleshooting Steps Commands
Create a monitor monitor capture mycap <interface | vlan | control-plane>
Add core filter monitor capture mycap [access-list <acl>| match <in-line match CLI>]
Display monitor details show monitor capture
Start/stop a monitor session monitor capture mycap start | stop

Display a pcap file show monitor capture file <filename>
Display a pcap file in detail show monitor capture file <filename>detailed
Display a pcap file with filter show monitor capture file <filename>display-filter filter-detail
Check if wireshark is running show proc cpu | inc dumpcap
96
Tools: Embedded Event Manager
Extremely versatile tool for monitoring, automating, working around issues
(a) What do I want to detect? (b) What do I want to do after that?

event manager appl et hi gh- cpu
event snmp oi d 1. 3. 6. 1. 4. 1. 9. 9. 109. 1. 1. 1. 1. 10. 1 get - t ype exact ent r y- op ge ent r y- val 80" pol l - i nt er val 10
act i on 1. 0 sysl og msg " HI GH_CPU! CPU i s at : $_snmp_oi d_val
act i on 2. 0 cl i command " enabl e"
act i on 2. 1 cl i command " show pr ocess cpu | r edi r ect boot f l ash: cpu. t xt "
act i on 2. 2 cl i command " conf i gur e t er mi nal "
act i on 2. 3 cl i command " event manager schedul er suspend

%HA_EM-6-LOG: TEST: HIGH_CPU! CPU is at: 99

event manager appl et i nt er f ace- f l appi ng
event sysl og pat t er n " . *UPDOWN. *Gi gabi t Et her net 1/ 1. *" occur s 4
act i on 1. 0 sysl og msg Gi gabi t Et her net I nt er f ace 1/ 1 changed st at e 4 t i mes
act i on 2. 0 cl i command " enabl e"
act i on 2. 2 cl i command " conf i gur e t er mi nal "
act i on 2. 3 cl i command i nt er f ace Gi gabi t Et her net 1/ 1
act i on 2. 4 cl i command shut down
Collect process CPU usage when CPU is high
Bring an interface down when it flaps too frequently
97
Embedded Event Manager / Netflow Integration

1. Packets with TTL=1 sent to the switch (TTL=1 streams can cause high CPU)
2. NetFlow Engine collects the flow capturing the TTL value:
%HA_EM-6-LOG: ttl: Flow Monitor ttl reported Low TTL for 10.10.10.3 10.10.10.4
3. EEM triggers a syslog when flow is detected:
swi t ch# sh runn flow record ttl
mat ch i pv4 t t l
mat ch i pv4 pr ot ocol
mat ch i pv4 sour ce addr ess
mat ch i pv4 dest i nat i on addr ess
col l ect count er byt es
col l ect count er packet s
col l ect t i mest amp sys- upt i me f i r st
col l ect t i mest amp sys- upt i me l ast
swi t ch# sh runn flow monitor ttl
Cur r ent conf i gur at i on:
f l ow moni t or t t l
r ecor d t t l
cache t i meout act i ve 40
swi t ch# sh runn int gi 6/1
no swi t chpor t
i p f l ow moni t or t t l i nput
i p addr ess 10. 10. 10. 2 255. 255. 255. 254
swi t ch(conf i g) # event manager applet ttl
event nf monitor-name "ttl"
event-type create event1 entry-value "2"
field ipv4 ttl entry-op lt

action 1.0 syslog msg
"Flow Monitor $_nf_monitor_name reported Low TTL
for $_nf_source_address $_nf_dest_address"
check show f l ow moni t or t t l cache f or mat r ecor d for IP TTL: 1
98
Tips: Crashes
Enhanced crashdump features in 15.0(2)SG2 / 3.2.2SG and higher
except i on cor edump highly recommended on IOS-XE
Classic IOS full core in 15.1(1)SG2 onwards
On IOS-XE, collect all files in crashinfo: and kinfo:
99
Tips: Miscellaneous
Enable NTP to troubleshoot across switches
Include date and time for debug and log messages
ser vi ce t i mest amps [ debug, l og] msec l ocal t i me show- t i mezone
Automatically output time and CPU utilization with each command (exec mode)
t er mi nal exec pr ompt t i mest amp
When logging the console, add comments and prefix with ! to avoid error messages
swi t ch#! ! ! show modul e af t er peer r el oad
swi t ch# show modul e
100
Tips: Make Life Easier
Search Bug Toolkit for known issues
Output Interpreter to decode command output
System Message Guide for mitigation recommendations
Smart Call Home in 12.2(52)SG
Catalyst 4000 Troubleshooting TechNotes
Catalyst 4500 Configuration Guide and Release Notes
NetPro discussion groups on http://www.cisco.com
101
Tips: Platform Control Plane Enhancements
Common Drop Event
Reason
First
Available
Control Packet Data
Plane Qos
12.2(54)SG Per-interface qos policies can drop control packets
Control Packet
Enhancements
15.0(2)SG /
3.2.0SG
Many static ACLs matching control traffic removed
CPU now included in special control floodsets on a per-vlan basis

access- l i st har dwar e capt ur e mode now controls only IGMP ACLs
CPU queue rate limits 15.1(1)SG /
3.3.0SG
DBL (per-flow rate limits) are applied to some CPU queues

Improved areas include:
port security / dot1x violate mode
non-RPF multicast (fast drop)

Drops appear as DblDrop in show pl at f or msof t war e dr op- por t

show pl at f or msof t war e i p mf i b f ast dr op deprecated
102
Maximize your Cisco Live experience with your
free Cisco Live 365 account. Download session
PDFs, view sessions on-demand and participate in
live activities throughout the year. Click the Enter
Cisco Live 365 button in your Cisco Live portal to
log in.
Complete Your Online Session Evaluation
Give us your feedback and
you could win fabulous prizes.
Winners announced daily.
Receive 20 Cisco Daily Challenge
points for each session evaluation
you complete.
Complete your session evaluation
online now through either the mobile
app or internet kiosk stations.
103

2013 - Usa - PDF - BRKCRS-3142 - Troubleshooting Cisco Catalyst 4500 Series Switches

Transféré par

Informations du document

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

2013 - Usa - PDF - BRKCRS-3142 - Troubleshooting Cisco Catalyst 4500 Series Switches

Transféré par

Droits d'auteur :

Formats disponibles

Troubleshooting Cisco Catalyst 4500 Series

%CPU %CPU RunTi meMax Pr i or i t y Aver age %CPU Tot al

Packet s Dr opped by Packet Queue

Super por t Tx Queue space di st r i but i on

40 1 10 0x05A558 0x05EED8 18841

0x00 Cont r ol Reg 0x1140

0x04 Aut oNegAdvReg 0x01E1

0x09 1000BaseTCont r ol Reg 0x0F00

Gi 7/ 48 35 4 210 8 402 Po1( 417) 367

swi t ch# show int vl 902 | i SVI

i nt er f ace Gi gabi t Et her net 3/ 19

i p access- l i st ext ended deny

Opcode : 40000 / 40000

CamI ndex Ent r y Type Act i ve Appl y QoS Hi t Count

I P Sr c : 1. 1. 1. 1 / 255. 255. 255. 255

Act I dx: 244 St at sI dx: 0 FwdI dx: ( Adj , Adj : 8)

Act I dx: 254 St at sI dx: 0 FwdI dx: ( None, r ep: 0)

000022: v4 192. 168. 200. 0/ 24 - - > vr f : Gl obal Rout i ng Tabl e ( 0)

Bl ock: 3 En: t r ue Ent r yMap: LSB Wi dt h: 80- Bi t Type: Sr c

012333: v4 91. 91. 91. 0/ 24 * - - > vr f : Gl obal Rout i ng Tabl e ( 0)

ReasonQueue i s not empt y

Phypor t TxQ Head Tai l Pr e Empt y Num BaseAddr Si ze Shape- Ok

Phypor t TxQ Head Tai l Pr e Empt y Num BaseAddr Si ze Shape- Ok

Execut i ng t he command on VSS member swi t ch r ol e = VSS

I pv4 Mul t i cast Fl oodset :

I pv4 Mul t i cast Fl oodset :

- Emer gency aged 0

Bucket s w/ X Bucket Count Used Ent r y Count

Vous aimerez peut-être aussi