BRKCRT 2077

Incident Handling for MPLS-Based Services
in Service Provider Environments

BRKCRT-2077
Contents
 Introduction
 Infrastructure and Services
 Troubleshooting Techniques
 Troubleshooting Core MPLS Issues
 Troubleshooting Layer-3 MPLS VPNs
 Troubleshooting Layer-2 MPLS VPNs
BRKCRT-2077 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2
Introduction
Incident Definition
 An incident is an event with a negative connotation

 Incidents in service provider networks are
caused by:
Hardware failures
Software failures
Design failures
Configuration failures
False positives
Incident Detection
Monitoring Alert Ticketing Issue End
Server System User
SNMP
Service
Desk
 Incidents detection options:

Monitoring systems (e.g. SNMP, Syslog)
Reported by customers
False negative (i.e. not detected)
 Ticketing systems are used to keep track of incident handling

procedures
Incident Severity
 Incident severity rating is proportional to:

Type of consequence
Scope
 Consequences of an incident:
Reduced level of resilience (e.g. lost backup link)
Reduced performance (e.g. lost member of port channel)
Loss of service
 Scope of an incident:
Physical (e.g. link vs. router)
Logical (e.g. single adjacency vs. entire protocol failure)
Geographical (e.g. local vs. global)
Design Characteristics to
Minimize Impact of Failures
Redundant Infrastructure:
 Redundant network devices
 Redundant control plane (e.g. route processors)
 Redundant data plane (e.g. redundant line cards)
 Redundant network connections
Design Characteristics to
Minimize Impact of Failures (Cont.)
Redundant Mechanisms:
 Routing protocols (routers)
 Spanning tree (LAN switches)
 Fast hellos in routing protocols or BFD for faster
convergence
 Link aggregation
 HSRP, VRRP, GLBP (typically at network edges)
 Stateful switchover (SSO) with NSF/NSR and
graceful restart
 MPLS Traffic Engineering with FastReroute
 Protected paths in optical networks (WDM)
Infrastructure and Services
Network Infrastructure
10Gbps 10Gbps Nx•
Router Router
Switch Router DWDM DWDM Router Switch
Network Devices Network Links

 Routers  Optical links
 LAN switches  *bitEthernet links
 WDM switches  SONET/SDH links
 Other  Other
Network Services
MP-BGP
IGP (OSPF, IS-IS)
LDP
Router Router
Switch Router DWDM DWDM Router Switch
 MP-BGP (Internet, L3 MPLS VPN, VPLS)

 IGP (OSPF, IS-IS)
 LDP (MPLS, L2 MPLS VPN, VPLS)
 RSVP (MPLS TE)
 Other
Redundant Device
Example: Cisco CRS-1
Designed Built-in Built-in
redundancy redundancy redundancy
Physical Layer Modular Services

Interface Module Card (MSC)
(PLIM) Switch Fabric Route Processor (RP)
Switch Fabric
Switch
Module Fabric
(SFM)
Switch Fabric Module Route Processor (RP)
Module (SFM)
Module
(SFM) (SFM)
Physical Layer Modular Services
(PLIM)
Automatic
state
Physical Layer Modular Services replication
(PLIM) Automatic
detection of
failed modules
Data Plane Control & Management Plane
Layer 3 MPLS VPNs
IP IP
Site 1 CE PE PE CE
Site 3
IP
+
IP MPLS IP
Site 2 CE PE PE CE
Site 4
VPN A
 Virtual routing and forwarding instances (VRFs)

used to contain VPN routing info
 MP-BGP used to exchange VPN routing info and
VPN labels
 Static routing, IGP or BGP used to exchange
routing info between PE and CE
Layer 3 MPLS VPNs
Topologies and Services
Central Services VPN
IP IP
Site 1 CE PE PE CE
Site 1
IP
+
IP MPLS IP
Site 2 CE PE PE CE
Site 2
PE PE
VPN A Central VPN B
Services
VPN
 MP-BGP uses route target (RT) extended

community attributes to control exporting and
importing of VPN routing info between VPNs
Layer 3 MPLS VPNs
Topologies and Services
Internet IP
Internet Management PSTN
Access Telephony
VPN
VPN VPN
IPTV Satellite
IP
VPN TV
+
MPLS
VPN A VPN B VPN C VPN D
 Internet access
 Centralized management of managed customer devices
 IP telephony
 IPTV
 Other
Layer 2 MPLS VPNs
Ethernet Ethernet
Site 1 Virtual Circuit Site 3
IP
+
MPLS
ATM Frame
Relay
 Two topologies:
Point to point
Point to multipoint
 Two major encapsulation options:

Same Layer 2 encapsulation on both ends
Interworking (translation from one Layer 2 encapsulation to another)
 Point-to-point Layer 2 virtual circuits across MPLS

 No need for IP peering and routing configuration
Point-to-Point Ethernet Over MPLS
Ethernet Ethernet
IP
+
MPLS
VLAN VLAN
 Two modes of operation

Port mode—entire Ethernet frames are encapsulated into
an MPSL LSP
VLAN mode—selected VLANs are extracted and
encapsulated into dedicate MPLS LSPs
VPLS
Ethernet Ethernet
IP + MPLS
Virtual Circuit
Virtual Circuit
Ethernet Ethernet
 Multipoint Ethernet over MPLS

 MPLS network is like a virtual switch
 Two implementation options:
Full mesh of end-to-end LDP sessions
BGP Auto Discovery
MPLS Traffic Engineering to
Optimize Resource Utilization
20%  50%
Head-end 40%  10% 80%  50% Tail-end
 Redundant networks may experience unequal load

in their network
 MPLS TE tunnels can be used to enable traffic flow
across any path
MPLS Traffic Engineering for
Improved Resilience
Backup tunnel for
protected node
Backup
Tunnel
Head-end Tail-end
Protected link
Protected node
 MPLS TE tunnels can be used to back up:

Links
Routers
 MPLS TE with FastReroute requires fast detection of failures:
Supported by underlying physical infrastructure or
Enabled through the use of BFD
 MPLS TE FastReroute implementation options:
Manual backup configuration
Auto-tunnel primary and backup configuration
Troubleshooting Techniques
Troubleshooting Procedures
 Troubleshooting should lead to the diagnosis and

the solution to a problem
Incident Diagnosis Solution
Problem Detection
 Problem can be detected:

By monitoring systems (SNMP, Syslog messages)
Reported by customer
Ticket

 Link state change
 IGP or BGP Adjacency change
Monitoring
system  IP SLA reported path failures
 Threshold exceeded:
On router (e.g. maximum number of VRF routes)
On monitoring system
Diagnostic Phases
 Diagnosis should lead to the nature and cause of a

problem
Information Information Hypotheses

gathering analyses proposition
Hypotheses Hypotheses
elimination testing
Troubleshooting Methods
Information Information Hypotheses

gathering analyses proposition
Hypotheses Hypotheses
elimination testing
 Iterate through the steps until a solution is found
Troubleshooting Methods
 The “shoot from the hip” method often used by experienced

operators and is based on previous experience
 The “follow the path” method is the most methodical way where you
trace the path of packets
 The “spot the difference” method is used to compare the setup with
those on working devices
 The “move the problem” method is used to detect hardware failures
by swapping components
 The “top-down” method used on an individual device referring to
OSI Layers starting from control plane down to data plane (physical
layer)
 The “bottom-up” method used on an individual device referring to
OSI Layers starting on the data plane (physical layer) up to control
plane
 The “divide and conquer” method used on an individual device
referring to OSI Layers starting somewhere in the middle (e.g. IP
routing table) and determining whether to move up or down
Troubleshooting Procedure
Example: Top Down
Fault Fault Fault

MP-BGP PE to PE Next hop Physical
adjacencies connectivity reachability layer
OK OK OK OK
Redistribution BGP IGP issues LDP issues

Route targets parameters Layer 2 issues
TCP filtering
Hardware Fault
problem
Example: Bottom Up
OK OK OK OK
Physical Next hop PE to PE MP-BGP
layer reachability connectivity adjacencies
Fault Fault Fault Fault
Hardware LDP issues IGP issues BGP

problem Layer 2 issues parameters
TCP filtering
Redistribution
Route targets
MPLS Data Structures and Terminology
IGP BGP LDP RSVP
(OSPF, IS-IS,
EIGRP)
RIB LIB
(LSD)
Control plane
Data plane
Core LSR and Egress
Edge LSR Functionality swap
LFIB
L IP (LFD) L IP
Ingress Edge
LSR Functionality
FIB
IP IP
route
Note! Cisco IOS and Cisco IOS XR may use slightly different terminology in
documentation
Note! The introduction of VRFs multiplies these data structures or creates
address families within them
Example: Follow the Path
MPLS Domain
Packet direction
10.1.1.5 35 65 10.1.1.5 43 43
65 10.1.1.5 65 10.1.1.5 10.1.1.5
Ingress PE P P Egress PE
LFIB LFIB
35  43; GigE3/1/0 43  pop; GigE1/1/4
CEF LFIB
10.1.1.0/24  L=35,65, GigE0/0/0 65  untag; GigE0/0/0 10.1.1.5
 Ingress PE router performs a routing lookup (CEF) and forwards a labeled packet:
Top label is for the BGP next-hop (e.g. loopback on egress PE)
Bottom label is a VPN label (e.g. identifies the outgoing interface on egress PE
 Core P routers perform label switching by swapping top labels

 Penultimate core P router removes (pops) the top label
 Egress PE forwards the packet based on the remaining label
Solution Implementation
Implement a
solution
Problem Diagnosis Solution

 Multiple solutions or workarounds can be found
 Chose the solution with: Document Close ticket
process
The lowest impact and/or
The lowest risk
 Mitigate the issue by implementing the chosen solution
 Disruptive solutions should be implemented during
maintenance windows unless the incident is critical
 Document the troubleshooting process and solution
 Close the ticket
Troubleshooting Core MPLS Issues
Verification Steps
BGP Routing
 Verify neighbor adjacency: Cisco IOS XR
RP/0/0/CPU0:PE1#show bgp vpnv4 unicast summary
...
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
2.2.2.2 4 65000 2099 2303 6 0 0 1d05h 2
 Possible states and causes are: Cisco IOS

Idle: Router#show bgp vpnv4 unicast all summary
...
No route to neighbor ... TblVer InQ OutQ Up/Down St/PfxRcd
... 0 0 0 00:00:00 Idle!
Active:
No route for return traffic
Traffic filtering
No neighbor configured on remote end
Verification Steps
BGP Routing – Idle State
1.1.1.1 2.2.2.2
 Usually local issue

RP/0/0/CPU0:PE1# show bgp vpnv4 unicast summary
...
Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
2.2.2.2 4 65000 0 0 0 0 0 00:00:00 Idle
 Verify reachability of the neighbor

RP/0/0/CPU0:PE1# show route ipv4 2.2.2.2
% Network not in table
 Check IGP settings if no route exists
Verification Steps
IGP Routing – Interfaces
 Verify that all core interfaces are included in IGP
In Cisco IOS XR Software:
show ospf interface brief
show isis interface brief
In Cisco IOS Software:
show ip ospf interface brief
show clns interface
 Loopbacks must also be included in IGP

Router#show ip ospf interface brief
Interface PID Area IP Address/Mask Cost State Nbrs F/C
Et1/3 1 0 10.177.67.1/16 10 DR 0/0
Lo1 1 0 1.1.1.1/32 1 LOOP 0/0
Et1/0 1 0 10.0.0.1/24 10 DR 0/0
Verification Steps
Routing Table
 Verify that routes for the MPLS network and all
neighbors are present
 If routes are missing, check routing protocol settings
on other routers
RP/0/0/CPU0:PE1#show route ipv4
...
Gateway of last resort is not set
i L2 1.1.1.1/32 [105/20] via 192.168.13.1, 1d02h, POS0/3/0/4

i L2 2.2.2.2/32 [105/20] via 192.168.23.2, 1d03h, POS0/3/0/1
O IA 10.1.1.0/24 [254/0] via 0.0.0.0, 4d03h, Null0
L 10.1.1.1/32 is directly connected, 1w0d, Loopback0
S 10.10.1.1/32 [1/0] via 192.168.111.2, 2d03h
S 10.100.12.0/24 [1/0] via 192.168.22.10, 08:29:56
L 127.0.0.0/8 [0/0] via 0.0.0.0, 1w0d
C 172.21.116.0/24 is directly connected, 1w0d, MgmtEth0/0/CPU0/0
L 172.21.116.10/32 is directly connected, 1w0d, MgmtEth0/0/CPU0/0
i L2 192.168.12.0/24 [105/20] via 192.168.23.2, 1d03h, POS0/3/0/1
[105/20] via 192.168.13.1, 1d03h, POS0/3/0/4
C 192.168.13.0/24 is directly connected, 1w0d, POS0/3/0/4
...
Verification Steps
LFIB Table
 View the table with show mpls forwarding-table or
show mpls forwarding
 Check if loopbacks of all routers are present in the
LFIB table
RP/0/0/CPU0:PE1#show mpls forwarding
Local Outgoing Prefix Outgoing Next Hop Bytes
Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
16000 Pop 1.1.1.1/32 PO0/3/0/4 192.168.13.1 464471
16001 Pop 192.168.33.0/24 PO0/3/0/1 192.168.23.2 0
16002 Pop 192.168.12.0/24 PO0/3/0/1 192.168.23.2 0
Pop 192.168.12.0/24 PO0/3/0/4 192.168.13.1 0
16003 Unlabelled 10.10.1.1/32 Gi0/4/0/3 192.168.111.2 561853
16006 Pop 2.2.2.2/32 PO0/3/0/1 192.168.23.2 191995
Verification Steps
Using Traceroute or LSP Traceroute
 Displays the packet path
 Uses UDP, TTL and ICMP to report the path
 Can report MPLS labels and BGP AS numbers
 Useful for:
Determining traffic paths
Localizing faults
 Only one path may be shown when load balancing

Router#traceroute 10.10.10.4
Type escape sequence to abort.

Tracing the route to 10.10.10.4
1 10.1.1.21 [MPLS: Label 20 Exp 0] 272 msec 268 msec 300 msec
2 10.1.1.5 [MPLS: Label 16 Exp 0] 228 msec 228 msec 228 msec
3 10.1.1.14 92 msec * 92 msec
Troubleshooting Layer-3 MPLS VPNs
Layer-3 MPLS VPNs Components
 Interface
 VRF
 MP-BGP routing protocol PE1
CE1
 Data Structures: MP-BGP
OSPF
 VRF address family of MP-BGP
 VPNv4 address family of MP-BGP
 Per-VRF or VRF address family PE-CE routing protocol
 VRF FIB
 LFIB
PE-CE Interface Misconfigurations
PE1
Fa0/0 Fa0/0
CE1 OSPF
CE1
Neighbor ID Pri State Dead Time Address Interface
10.1.1.1 1 INIT/DROTHER 00:00:39 10.1.1.1 FastEthernet0/0
 Disabled interface
 Misconfigured IP address
 Access list denying routing protocol traffic
 Oversized MTU:
May require manual configuration due to ICMP traffic filtering
PE-CE Routing Misconfigurations
PE1
interface FastEthernet0/0 Fa0/0 Fa0/0 interface FastEthernet0/0

mtu 1520 mtu 1500
CE1 OSPF
Neighbor ID Pri State Dead Time Address Interface

10.1.1.1 1 INIT/DROTHER 00:00:39 10.1.1.1 FastEthernet0/0
 Neighborship cannot be formed due to MTU mismatch:

Exact MTU match required for OSPF neighbors
Few bytes difference permitted by IS-IS
 Route filter denying routing information
 Limited number of received prefixes from neighbor
PE-CE Routing Misconfigurations
(Cont.)
OSPF: Rcv pkt from 10.1.1.2, FastEthernet0/0 : Mismatch Authentication

type. Input packet specified type 2, we use type 0
OSPF: Rcv pkt from 10.1.1.2, FastEthernet0/0 : Mismatch Authentication

Key - Message Digest Key 0
 Mismatched authentication credentials

 Mismatched interface type with OSPF or IS-IS
%BGP-3-NOTIFICATION: received from neighbor 10.1.1.1 2/2 (peer in wrong

AS) 2 bytes 0001
 BGP configuration mismatch:

Wrong BGP AS number
Disabled routing neighbor
PE Specific Misconfigurations
P
 Interface:
MTU - no support for jumbo frames P
Disabled CEF PE
Incorrect VRF membership
%IPRT-3-ROUTELIMITEXCEEDED : IP routing table limit exceeded – CE1,

172.16.99.0/24
 VRF:
Low limit for total number of imported routes
Misconfigured import or export
PE Specific Misconfigurations (Cont.)
RT 1:11 PE1 PE2

OSPF
RT 1:11
OSPF PE3
RT 1:99
OSPF
 MP-BGP:
Remote sites inaccessible due to IGP routes not redistributed
Mismatched route targets break connectivity for sites of the
same VPN
PE-CE Routing Verification (Cont.)
 BGP routing
Verify neighbor adjacency on PE and CE routers
Idle or Active state – neighbor reachability
Cisco IOS XR
RP/0/RP0/CPU0:PE1#show bgp vrf VPNA summary
BGP VRF VPNA, state: Active
BGP Route Distinguisher: 1:100
...
Process RecvTblVer bRIB/RIB LabelVer ImportVer SendTblVer
Speaker 107 107 107 107 107
...
Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd
10.1.101.2 0 65001 1601 1464 107 0 0 1d00h 4
...
Cisco IOS
CE1#show ip bgp summary
BGP router identifier 10.9.2.1, local AS number 1
BGP table version is 104, main routing table version 104
29 network entries using 3509 bytes of memory
29 path entries using 1508 bytes of memory
...
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.101.1 4 1 1460 1601 104 0 0 1d00h 29
...
MP-BGP Routing Verification (Cont.)
 Verify MP-BGP table for customer VPN prefixes:

Examine originated and received VPNv4 prefixes
Cisco IOS XR
RP/0/RP0/CPU0:PE1#show bgp vrf VPNB
...
BGP generic scan interval 60 secs
BGP table state: Active
BGP main routing table version 167
BGP scan interval 60 secs
Status codes: s suppressed, d damped, h history, * valid, > best
i - internal, S stale
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:200 (default for vrf VPNB)
*> 10.1.1.0/24 0.0.0.0 0 32768 ? Locally-originated or
*> 10.1.99.1/32 0.0.0.0 0 32768 ? redistributed VPNv4 prefixes
*>i10.177.0.0/16 172.16.1.1 0 100 0 ?
*>i10.1.92.1/32 172.16.1.1 0 100 0 ? Received VPNv4 Prefixes
...
VPN Label Verification
 Verfiy VPN labels for customer VPN prefixes:

Examine locally originated and remote VPN labels
Cisco IOS XR
RP/0/RP0/CPU0:PE1#show bgp vpnv4 unicast labels
...
BGP scan interval 60 secs
Status codes: s suppressed, d damped, h history, * valid, > best
i - internal, S stale
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Rcvd Label Local Label
Route Distinguisher: 1:200 (default for vrf VPNA)
*> 10.1.1.0/24 0.0.0.0 nolabel 16024 Locally-originated VPN labels
*> 10.1.99.1/32 0.0.0.0 nolabel 16034
*>i10.177.0.0/16 172.16.1.1 38 nolabel
*>i10.1.92.1/32 172.16.1.1 18 nolabel Received VPN labels
...
VPN Label Verification (Cont.)
 Verfiy individual prefix VPN labels:

Examine originated VPNv4 prefixes
Cisco IOS XR
RP/0/RP0/CPU0:PE1#show bgp vrf VPNA 10.1.1.0
BGP routing table entry for 10.1.1.0/24, Route Distinguisher: 23456:100
Versions:
Process bRIB/RIB SendTblVer
Speaker 2 2
Local Label: 16024
Paths: (1 available, best #1)
Advertised to PE peers (in unique update groups):
172.16.1.1
Path #1: Received by speaker 0
Local
0.0.0.0 from 0.0.0.0 (10.1.101.2)
Origin incomplete, metric 0, localpref 100, weight 32768, valid, redistributed,
best, import-candidate
Extended community: RT:23456:100 OSPF route-type:0:2:0x0 OSPF router-
id:10.1.99.1
VPN Label Verification (Cont.)
 Verfiy individual prefix VPN labels:

Examine recieved VPNv4 prefixes
Cisco IOS XR
RP/0/RP0/CPU0:PE1#show bgp vrf VPNA 10.177.0.0/16
BGP routing table entry for VPNA 10.177.0.0/16, Route Distinguisher: 23456:100
Versions:
Process bRIB/RIB SendTblVer
Speaker 123 112
Paths: (1 available, best #1)
Not advertised to any peer
Path #1: Received by speaker 0
Local
10.1.4.1 (metric 3) from 10.1.3.1 (10.1.4.1)
Received Label 38
Origin incomplete, metric 0, localpref 100, valid, internal, best, import-
candidate, imported
Extended community: 23456:100
Originator: 10.1.4.1, Cluster list: 10.8.11.1
CEF Table Verification
 Examine FIB table for customer VPN prefix:

Verify that the packets entering the MPLS get proper labels
Top label – LDP label
Bottom label – VPN label
Cisco IOS XR
RP/0/RP0/CPU0:PE1#show cef vrf VPNA 10.177.0.0
10.177.0.0/16, version 633, internal 0x40040001[1]
Prefix Len 16, traffic index 0, precedence routine (0)
via 172.16.1.1, 0 dependencies, recursive
next hop 172.16.1.1 via 172.16.1.1 /32
next hop 10.1.102.1 Gi0/1/0/6 labels imposed {20 38}
LDP label VPN label
Troubleshooting Layer-2 MPLS VPNs
Verify VC Parameters and Status
MPLS Domain
LDP or MP-BGP
Layer 2 IGP IGP IGP Layer 2
Frame LDP LDP LDP Frame
Edge Core Core Edge

LSR LSR LSR LSR
Interface status and VC status and VC status and Interface status and
configuration configuration configuration configuration
 Check status and configuration of main

components:
Customer-facing interfaces
VCs
MPLS Domain
LDP or MP-BGP
Edge Core Core Edge

LSR LSR LSR LSR
Cisco IOS
PE1# show mpls l2transport vc 10
Local intf Local circuit Dest address VC ID Status

------------- -------------------------- --------------- ---------- ----------
Se1/0 HDLC 172.16.1.4 10 UP
PE1#
Cisco IOS
PE4# show mpls l2transport vc 10

------------- -------------------------- --------------- ---------- ----------
Se1/0 HDLC 172.16.1.1 10 UP
PE4#
MPLS Domain
LDP or MP-BGP
Edge Core Core Edge

LSR LSR LSR LSR
Cisco IOS
PE1# show mpls l2transport binding PE4# show mpls l2transport binding
Destination Address: 172.16.1.4, VC ID: 10 Destination Address: 172.16.1.1, VC ID: 10
Local Label: 20 Local Label: 21
Cbit: 1, VC Type: HDLC, GroupID: 0 Cbit: 1, VC Type: HDLC, GroupID: 0
MTU: 1500, Interface Desc: n/a MTU: 1500, Interface Desc: n/a
VCCV: CC Type: CW [1], RA [2] VCCV: CC Type: CW [1], RA [2]
CV Type: LSPV [2] CV Type: LSPV [2]
Remote Label: 21 Remote Label: 20
PE1# PE4#
Verify Data Plane on Egress PE
Forwarding From AC to VC – Cisco IOS
Layer 2 MPLS Domain Layer 2

Frame VC Frame
Cisco IOS Edge Core Core Edge

LSR
PE2# show mpls l2transport LSR FastEthernet1/1
vc interface LSR detail LSR
Local interface: Fa1/1 up, line protocol up, Ethernet up
Destination address: 172.16.1.11, VC ID: 1101, VC status: up
Output interface: Fa0/0, imposed label stack {17 28}  Interfaces status “up”
Preferred path: not configured
Default path: active  Presence of label
Next hop: 192.168.1.5
Create time: 01:42:52, last status change time: 01:14:36 stack
Signaling protocol: LDP, peer 172.16.1.11:0 up
Targeted Hello: 172.16.1.12(LDP Id) -> 172.16.1.11  Non-zero statistics
Status TLV support (local/remote) : enabled/supported
Label/status state machine : established, LruRru for sent and received
... packets
MPLS VC labels: local 28, remote 28
Group ID: local 0, remote 0
MTU: local 1500, remote 1500
VC statistics:
packet totals: receive 5, send 719
byte totals: receive 516, send 93412
packet drops: receive 0, seq error 0, send 0
Forwarding From AC to VC – Cisco IOS XR
MPLS Domain
Layer 2 Layer 2
Frame VC Frame
Edge Core Core Edge

LSR LSR LSR LSR
Cisco IOS XR
RP/0/RP1/CPU0:CRS# show l2vpn forwarding detail location 0/1/CPU0
Local interface: GigabitEthernet0/1/0/2.2, Xconnect id: 1101, Status: up
Segment 1
AC, GigabitEthernet0/1/0/2.2, Ethernet VLAN mode, status: Bound
Packet switched: 22, byte switched: 2272
Packet dropped (MTU/Queue): 0/0, byte dropped (MTU/Queue): 0/0
Segment 2
AToM, Destination address: 10.2.3.1, status: Bound
MPLS pseudowire label: 26
Packet switched: 28, byte switched: 2656
Packet dropped (PLU/Queue): 0/0, byte dropped (PLU/Queue): 0/0
RP/0/RP1/CPU0:CRS#
Forwarding From VC to AC – Cisco IOS
MPLS Domain
Layer 2 Layer 2
Frame VC Frame
Edge Core Core Edge

LSR LSR LSR LSR
Cisco IOS
PE2# show mpls forwarding-table
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or VC or Tunnel Id Switched interface
21 Pop Label 192.168.1.0/30 0 Fa0/0 192.168.1.5
22 Pop Label 192.168.1.8/30 454 Fa0/0 192.168.1.5
Pop Label 192.168.1.8/30 0 Fa0/1 192.168.1.25
23 Pop Label 192.168.1.12/30 0 Fa0/0 192.168.1.5
24 Pop Label 192.168.1.16/30 4534 Fa0/0 192.168.1.5
25 Pop Label 192.168.1.20/30 55445 Fa0/1 192.168.1.25
26 Pop Label 192.168.1.28/30 0 Fa0/1 192.168.1.25
27 Pop Label 192.168.1.32/30 1234 Fa0/1 192.168.1.25
28 No Label l2ckt(1101) 212343 Fa1/1 point2point
...
PE2#
Forwarding From VC to AC – Cisco IOS XR
MPLS Domain
Layer 2 Layer 2
Frame VC Frame
Edge Core Core Edge

LSR LSR LSR LSR
Cisco IOS XR
RP/0/RP1/CPU0:CRS# show mpls forwarding
Local Outgoing Prefix Outgoing Next Hop Bytes
Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
...
16106 Unlabelled PW(10.2.3.1:1101) Gi0/1/0/2.2 point2point 233312
...
RP/0/RP1/CPU0:CRS#
Other Cisco IOS XR Commands
Cisco IOS XR
RP/0/RP1/CPU0:CRS# show l2vpn xconnect
Legend: ST = State, UP = Up, DN = Down, AD = Admin Down, UR = Unresolved,
LU = Local Up, RU = Remote Up, CO = Connected
XConnect Segment 1 Segment 2

Group Name ST Description ST Description ST
------------------------ ------------------------- -------------------------
LAN3 LAN3 UP Gi0/1/0/2 UP 10.2.3.1 1101 UP
--------------------------------------------------------------------------------
RP/0/RP1/CPU0:CRS#
Other Cisco IOS XR Commands
Cisco IOS XR
RP/0/RP1/CPU0:CRS# show l2vpn xconnect detail
Group LAN3, XC LAN3, state is up
Segment 1 is AC: GigabitEthernet0/1/0/2, Xconnect ID: 8, type Ethernet, MTU 1500
State is up
Statistics:
packet totals: send 5
byte totals: send 516
Segment 2 is PW: neighbor 10.2.3.1, PW ID 3
State is up ( established )
PW type Ethernet, interworking none
Encapsulation MPLS, protocol LDP
Label: local 16101, remote 26
Group ID: local 0x1180080, remote unknown
Control word: local set, remote set
Interface: local GigabitEthernet0/1/0/2, remote: unknown
MTU: local 1500, remote 1500
VCCV local CV Type = 0x2 (LSP ping verification)
VCCV local CC Type = 0x3 (control word, router alert label)
VCCV remote CV Type: 0x2 (LSP ping verification)
VCCV remote CC Type = 0x3 (control word, router alert label)
Create time: 23/01/2009 13:53:47 (03:01:31 ago)
Last time status changed: 23/01/2009 15:13:46 (01:41:32 ago)
Statistics:
packet totals: receive 2316
byte totals: receive 1205520
RP/0/RP1/CPU0:CRS#
Mismatched VC ID
Cisco IOS
Remote Label: unassigned Remote Label: unassigned
PE1# PE4#
 No VC created
 Correct configuration
Mismatched VC Type
Cisco IOS
PE1# show mpls l2transport vc

------------- -------------------------- --------------- ---------- ----------
Fa1/0.101 Eth VLAN 101 172.16.1.12 1101 DOWN
PE1#
PE2# show mpls l2transport vc

------------- -------------------------- --------------- ---------- ----------
Fa1/1 Ethernet 172.16.1.11 1101 DOWN
PE2#
 Check VC type on both PE routers

 Both ends of VC must be identical
 Solutions:
Ensure identical configuration
Configure interworking to bridge the two access methods
Mismatched VC Type - Interworking
MPLS Domain
802.1q Trunk Access

VLAN 101 Port
Edge Core Core Edge

LSR LSR LSR LSR
Cisco IOS
pseudowire-class VLAN2Eth interface FastEthernet1/1
encapsulation mpls no ip address
interworking ethernet xconnect 172.16.1.11 1101 encapsulation
! mpls
interface FastEthernet1/0.101 !
no ip address
description VLAN 101 to AC
encapsulation dot1Q 101
xconnect 172.16.1.12 1101 pw-class VLAN2Eth
!
 Enable interworking if required
Mismatched MTU
Cisco IOS
PE1# PE4#
 Check configuration of customer-facing interfaces

 Correct interface MTU
Core MTU Issues
MPLS Domain
1500-byte Ping
1500-byte MPLS Ping
Layer 2 Layer 2
Frame MTU=9200 MTU=9200 MTU=9200 Frame
Edge Core Core Edge

LSR LSR LSR LSR
Cisco IOS
PE1# show mpls interfaces GigabitEth2/2/0 detail CE4# ping 192.168.1.1 size 1500 df-bit
Interface GigabitEthernet2/2/0: Type escape sequence to abort.
IP labeling enabled (ldp): Sending 5, 1500-byte ICMP Echos to
Interface config 192.168.1.1, timeout is 2 seconds:
LSP Tunnel labeling enabled Packet sent with the DF bit set
BGP labeling not enabled !!!!!
MPLS operational Success rate is 100 percent (5/5),
MTU = 9200 round-trip min/avg/max = 12/32/56 ms
PE1# CE4#
 Customers typically require MTU 1500

 Core interfaces require more than 1500 (MPLS labels, control words, frame header)
 Typically not an issue with Gigabit and TenGigabit interfaces
 Alternatively, increase core MTU by 30 or more
Mismatched Labels
Cisco IOS
PE1# PE4#
 Most probably a software bug

 Collect information about the issue
 Restart targeted LDP session for a possible temporary fix
PE1# clear mpls ldp neighbor 172.16.1.4
 Open TAC case and provide troubleshooting info

Test VC Using MPLS Ping
MPLS Domain
MPLS Ping PE Loopback
Layer 2 MPLS Ping VC Layer 2

Frame Frame
Edge Core Core Edge

LSR LSR LSR LSR
 VC LSP depends on one or more other LSPs:

Hop-by-hop LDP
MPLS TE tunnels
 Use MPLS Ping and Traceroute to verify reachability

of egress PE loopback
 Also check for multiple paths
Egress PE Loopback Reachability
MPLS Domain
1500-byte Ping
Layer 2 Layer 2
Edge Core Core Edge

LSR LSR LSR LSR
Cisco IOS & Cisco IOS XR
PE1# ping mpls ipv4 172.16.1.4 255.255.255.255 size 1500 reply mode router-alert
Sending 5, 100-byte MPLS Echos to 172.16.1.4/32,
timeout is 2 seconds, send interval is 0 msec:
Codes: '!' - success, 'Q' - request not sent, '.' - timeout,

'L' - labeled output interface, 'B' - unlabeled output interface,
'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
'P' - no rx intf label prot, 'p' - premature termination of LSP,
'R' - transit router, 'I' - unknown upstream index,
'X' - unknown return code, 'x' - return code 0

!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/16/20 ms
PE1#
Test VC Using MPLS Ping
MPLS Domain
1500-byte Ping
Layer 2 Layer 2
Edge Core Core Edge

LSR LSR LSR LSR
Cisco IOS & Cisco IOS XR
PE1# ping mpls pseudowire 172.16.1.4 10 size 1500
Sending 5, 100-byte MPLS Echos to 172.16.1.4,
timeout is 2 seconds, send interval is 0 msec:
Codes: '!' - success, 'Q' - request not sent, '.' - timeout,

'L' - labeled output interface, 'B' - unlabeled output interface,
'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
'P' - no rx intf label prot, 'p' - premature termination of LSP,
'R' - transit router, 'I' - unknown upstream index,
'X' - unknown return code, 'x' - return code 0

!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/20/48 ms
PE1#
VPLS – Manual Configuration
 Lack of full mesh configuration with built-in split

horizon rules results in partial connectivity:
Check configuration of full mesh of LDP sessions
Check the status of all LDP sessions
Check the forwarding (MPLS Ping)
VPLS – Auto-Discovery
Route
IPv4 Reflector IPv4
VPLS VPLS
IPv4 IPv4
VPLS No VPLS
 VPLS relies on MP-BGP to carry VPLS information

throughout the network
 Make sure VPLS L2VPN address family is enabled on all
IBGP sessions
 Make sure extended BGP communities are forwarded on all
IBGP sessions
Hardware Capabilities
MPLS Domain
1500-byte Ping
Layer 2 Layer 2
Edge Core Core Edge

LSR
PE1# show mpls l2transport LSR
hw-capability LSR FastEthernetLSR
interface 1/0
Interface FastEthernet1/0
Transport type Eth VLAN

Core functionality:
MPLS label disposition supported
Control word processing supported
Sequence number processing supported
VCCV CC Type CW [1] processing supported
Edge functionality:
MPLS label imposition supported
Sequence number processing supported
Transport type Ethernet

Core functionality:
MPLS label disposition supported
...
Samples of Unsupported Features
 EoMPLS cannot be combined with private VLANs

 VLANs reuse depends on platform:
Unique VLANs are required on the Cisco 7600 series
platform unless ES modules are used
Cisco IOS XR allows reuse on VLANs
 EoMPLS only works with physical interfaces and

subinterfaces (not VLAN interfaces)
 VPLS is only supported Cisco IOS XR routers and
Cisco 7600 series routers
Summary
 Use a structured troubleshooting approach

 Document troubleshooting steps
 Remember that MPLS and VPNs add a number of
new data structures
 Determine the criticality of an incident
 Determine the disruptiveness of remedial actions
 Choose the least disruptive solution
 Use a maintenance window for disruptive solutions
unless the incident is critical
Q&A
Complete Your Online
Session Evaluation
 Give us your feedback and you

could win fabulous prizes.
Winners announced daily.
 Receive 20 Cisco Preferred
Access points for each session
evaluation you complete.
 Complete your session
evaluation online now (open a
browser through our wireless
network to access our portal)
or visit one of the Internet Don’t forget to activate your
stations throughout the Cisco Live and Networkers Virtual
Convention Center. account for access to all session
materials, communities, and on-demand
and live activities throughout the year.
Activate your account at any internet
station or visit www.ciscolivevirtual.com.
Enter to Win a 12-Book Library
of Your Choice from Cisco Press
Visit the Cisco Store in the

World of Solutions, where
you will be asked to enter
this Session ID code
Check the Recommended Reading brochure for

suggested products available at the Cisco Store

BRKCRT 2077

Transféré par

Informations du document

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

BRKCRT 2077

Transféré par

Droits d'auteur :

Formats disponibles

Incident Handling for MPLS-Based Services

in Service Provider Environments

 An incident is an event with a negative connotation

 Incidents detection options:

 Ticketing systems are used to keep track of incident handling

 Incident severity rating is proportional to:

10Gbps 10Gbps Nx•

Network Devices Network Links

IGP (OSPF, IS-IS)

 MP-BGP (Internet, L3 MPLS VPN, VPLS)

Physical Layer Modular Services

Data Plane Control & Management Plane

 Virtual routing and forwarding instances (VRFs)

 MP-BGP uses route target (RT) extended

VPN A VPN B VPN C VPN D

 Two major encapsulation options:

 Point-to-point Layer 2 virtual circuits across MPLS

 Two modes of operation

 Multipoint Ethernet over MPLS

Head-end 40%  10% 80%  50% Tail-end

 Redundant networks may experience unequal load

 MPLS TE tunnels can be used to back up:

 Troubleshooting should lead to the diagnosis and

Incident Diagnosis Solution

 Problem can be detected:

Incident Diagnosis Solution

 Diagnosis should lead to the nature and cause of a

Information Information Hypotheses

Incident Diagnosis Solution

Information Information Hypotheses

 Iterate through the steps until a solution is found

 The “shoot from the hip” method often used by experienced

Fault Fault Fault

Redistribution BGP IGP issues LDP issues

Fault Fault Fault Fault

Hardware LDP issues IGP issues BGP

 Core P routers perform label switching by swapping top labels

Problem Diagnosis Solution

 Possible states and causes are: Cisco IOS

 Usually local issue

 Verify reachability of the neighbor

% Network not in table

 Check IGP settings if no route exists

 Loopbacks must also be included in IGP

i L2 1.1.1.1/32 [105/20] via 192.168.13.1, 1d02h, POS0/3/0/4

 Only one path may be shown when load balancing

Type escape sequence to abort.

interface FastEthernet0/0 Fa0/0 Fa0/0 interface FastEthernet0/0

Neighbor ID Pri State Dead Time Address Interface

 Neighborship cannot be formed due to MTU mismatch:

OSPF: Rcv pkt from 10.1.1.2, FastEthernet0/0 : Mismatch Authentication

OSPF: Rcv pkt from 10.1.1.2, FastEthernet0/0 : Mismatch Authentication

 Mismatched authentication credentials

%BGP-3-NOTIFICATION: received from neighbor 10.1.1.1 2/2 (peer in wrong

 BGP configuration mismatch:

Incorrect VRF membership

%IPRT-3-ROUTELIMITEXCEEDED : IP routing table limit exceeded – CE1,

RT 1:11 PE1 PE2

 Verify MP-BGP table for customer VPN prefixes:

 Verfiy VPN labels for customer VPN prefixes:

 Verfiy individual prefix VPN labels:

 Verfiy individual prefix VPN labels:

 Examine FIB table for customer VPN prefix: