CEN Fault Isolation

Contents

1.

Checking Services using OTNM2000 GUI

2.

Checking Services using Command Line

Interface (CLI)

3.

Fault Isolation

Contents

1.

Checking Services using OTNM2000 GUI

2.

Checking Services using Command Line

Interface (CLI)

3.

Fault Isolation

Checking Services using OTNM2000 GUI

1. The Task Panel

ALARM: To view the current alarms

ALARM_HIS: To view the historical alarms

PERFORMANCE: To view NE current performance (e.g.

IOP(Input Optical Power/Received) and OOP(Output Optical
Power/Transmit), Rx/Tx , NE/FAN Temperature, Laser Bias,
Voltage, Memory Utilization

PERFORMANCE_HIS: Querying Performance Code

STATUS: To view the current port Link State, Optical Module

Information, Received and Transmitted Packets.

Card Configuration: FAN Speed Configuration, Clock Config.

MASK SETTING: Alarm and Performance masking/hiding

Control Command: Loopback Testing, Card Soft Reset, Card

Switching to Main/Protection, Port Laser ON/Off,

Checking Services using OTNM2000 GUI

2. The Help Toolbar

Alarm and Performance Help: Views the relevant information of the alarm
and performance.

Checking Services using OTNM2000 GUI

ALARM

To view the current and

timestamp of alarm: Right
click at the NE and select
Open. Click the alarmed
card.

Click the ALARM icon found

on the Task Panel.

Checking Services using OTNM2000 GUI

ALARM_HIS
Users can know about the faults of
network via viewing alarm history, so
as to facilitate the fault analysis.

Checking Services using OTNM2000 GUI

PERFORMANCE
Users can know whether the equipment or service in the network is abnormal or not
by viewing performances. Viewing performance also help the maintainers to handle
the faults in a timely manner and prevent faults.

Checking Services using OTNM2000 GUI

PERFORMANCE_HIS
View the performance history to obtain the abnormal performance data, in order to
instruct the current routine operations.

Checking Services using OTNM2000 GUI

STATUS

Via viewing card status, users can

check whether the cards are
present and the software /
hardware version complies with
the planning.

10

Checking Services using OTNM2000 GUI

Card Configuration
Via card configuration, users can enable E1 ports, configure clock and time
synchronization.

11

Checking Services using OTNM2000 GUI

Control Command
Via control command, users can perform software reset, card forced switch, E1
loop testing and laser on/off of optical ports.

12

Contents

1.

Checking Services using OTNM2000 GUI

2.

Checking Services using Command Line

Interface (CLI)

3.

Fault Isolation

13

Checking Services of Dynamic Networking

Check IS-IS, OSPF, and BGP neighbor establishment.

Check RSVP and LDP session setup.

Check whether BGP distributes direct routes.
Check VC establishment.

Check the BFD status.

Check the MPLS and VRF ILM forwarding tables.
Check services through LSP ping, VRF ping, and Internet Control
Message Protocol (ICMP) ping.

14

Service Self-test
Perform service self-test after service configuration.
OSPF check command:
router#show ip os ne
Output:
OSPF process 100:
Neighbor ID Pri State
Dead Time Address
Interface Instance
ID Up Time
6.6.6.6
1 Full/ 00:00:30 56.1.1.2
XGE0/6/1.31 0
01d00h00m
22.22.22.22
1 Full/DR
00:00:34 115.1.1.2
GE0/11/3.31 0
01d00h31m

15

Service Self-test
Perform service self-test after service configuration.
IS-IS check command:
router#show clns is-neighbors
Output:
Area 300:
System Id
Time
0101.1213.6185
83d05h51
0101.1213.6008
12d19h58

Interface State Type Priority Circuit Id

Up

XGE0/3/1

Up

L1 64

0101.1213.6185.03

XGE0/4/1

Up

L1 64

0101.1213.6186.03

16

Service Self-test
Check RSVP and LDP session setup.
Command:
router#show rsvp session
Output:
Ingress RSVP:
To
From
State Pri Rt Style Labelin Labelout LSPName
DSType UpTime
1.1.1.1
5.5.5.5
Up Yes 1 1 SE
- 131587 1-5.1-1.1.1e00000f ELSP_SIG 00:01:32
1.1.1.1
5.5.5.5
Up No 1 1 SE
- 131584 1-5.1-1.1.1e00000f ELSP_SIG 00:02:26
Total 2 displayed, Up 2, Down 0.
Egress RSVP:
To
From
State Pri Rt Style Labelin Labelout LSPName
DSType UpTime
5.5.5.5
1.1.1.1
Up Yes 1 1 SE 131589
- 1-5.1-1.1.1e00000f ELSP_SIG 00:01:30
5.5.5.5
1.1.1.1
Up Yes 1 1 SE 131585
- 1-5.1-1.1.1e00000f ELSP_SIG 00:02:17
Command:
router#show ldp session
Output :
Peer IP Address IF Name My Role State
KeepAlive Up Time
8.8.8.8
GE0/11/3.31Passive OPERATIONAL 60
01d00h44m
6.6.6.6
XGE0/6/1.31Passive OPERATIONAL 60
01d00h12m
22.22.22.22
GE0/11/3.31Passive OPERATIONAL 60
01d00h44m

17

Service Self-test
Perform service self-test after service configuration.
Check VC establishment.
Command:
router#show mpls vc-table
Output:
VC-Name
VC-ID Access-Intf Network-Intf In Label Out Label Tunnel-Label Nexthop
Status Oam-Status Up-Time Y1731
1-8.1-5.1
1
GE0/7/3.32 GE0/7/1.31 8713
132364 9219
5.5.5.5
Inactive NULL
00:49:30 0
1-8.1-6.2
2
GE0/7/3.32 GE0/7/2.31 8715
132096 9223
6.6.6.6
Active Bfd-UP
00:49:30 0
1-8.1-5.3
3
GE0/7/3.33 GE0/7/1.31 8714
132365 9216
5.5.5.5
Active Bfd-UP
00:49:30 0
1-8.1-6.4
4
GE0/7/3.33 GE0/7/2.31 8718
132097 9224
6.6.6.6
Standby Bfd-UP
01d20h52m 0

18

Service Self-test
Perform service self-test after service configuration.

Check the BFD status.

Command:
router#show bfd session

Output:
Sess-Idx Remote-Disc Lower-Layer Sess-Type Sess-State UP-Time Remote-Addr
1801
1806
IPv4
Single-Hop Up
1d01h00m 45.1.1.1/32
1803
802
IPv4
Single-Hop Up
1d01h00m 115.1.1.2/32
1
1
MPLS LSP Single-Hop Up
1d00h30m 127.0.0.12/32
1804
0
IPv4
Single-Hop Down
00:00:00 56.1.1.2/32
5
2
MPLS LSP Single-Hop Up
1d00h28m 127.0.0.12/32
4
1
MPLS LSP Single-Hop Up
1d00h28m 127.0.0.12/32
7
6
MPLS VCCV Single-Hop Up
00:53:30 127.0.0.12/32

19

Service Self-test
Perform service self-test after service configuration.
Check the MPLS and forwarding tables.
Command:
router#show mpls forwarding-table
Output:
Codes: > - selected FTN, p - stale FTN, B - BGP FTN, K - CLI FTN,
L - LDP FTN, R - RSVP-TE FTN, S - SNMP FTN, I - IGP-Shortcut,
U - unknown FTN
Code FEC
Tunnel-id FTN-ID Pri Nexthop
Out-Label Out-Intf LSP-Type LSPID Ingress-IDUp-Time
R>
1.1.1.1/32
1
1
Yes 45.1.1.1
131587
XGE0/6/4.31 1
5.5.5.5
ELSP_SIGNAL 00:25:54
R
1.1.1.1/32
1
20
No 56.1.1.2
131592
XGE0/6/1.31 2
5.5.5.5
ELSP_SIGNAL 00:07:47
R>
1.1.1.1/32
5
17
Yes 45.1.1.1
131588
XGE0/6/4.31 1
5.5.5.5
ELSP_SIGNAL 00:27:04
R>
2.2.2.2/32
2
19
Yes 56.1.1.2
131593
XGE0/6/1.31 1
5.5.5.5
ELSP_SIGNAL 01d00h31m
L>
6.6.6.6/32
0
2
Yes 56.1.1.2
132102
XGE0/6/1.31 LSP_DEFAULT
01d00h20m

20

Service Self-test
Perform service self-test after service configuration.
Command: FEC to Next Hop Label Forwarding Entry Table
router#show mpls ftn-table
Output:
Primary FTN entry with FEC: 1.1.1.1/32, id: 1, row status: Active
Owner: RSVP, Action-type: Redirect to Tunnel, Exp-bits: 0x0, Incoming DSCP: none
Tunnel id: 1, Protected LSP id: 1, QoS Resource id: 71, Description: 1-5.1-1.1.1e00000f
Select: 0
, Bfd id:2, MPLS-TP OAM Y1731 Initiative id:0, Lock:0 Cross connect ix: 290, in intf: - in
label: 0 out-segment ix: 269
Owner: RSVP, Persistent: No, Admin Status: Up, Oper Status: Up
Out-segment with ix: 269, owner: RSVP, out intf: XGE0/6/4.31, out label: 131587
Nexthop addr: 45.1.1.1 cross connect ix: 290, op code: Push
LSP Type: ELSP_SIGNAL
Class_Exp mapping:
CLASS DSCP_value EXP_value
be
000000
0
Bidirectional Forwarding Detection is configured
Bidirectional Forwarding Detection is enabled
Up Time: 00:26:05

21

Service Self-test
Perform service self-test after service configuration.
Command: MPLS Incoming Label Map
router#show mpls ilm-table
Output:
Codes: > - selected ILM, p - stale ILM, K - CLI ILM
Code In-Label Out-Label In-Intf
Out-Intf
Nexthop
FEC
ILM-ID
LSP-Type Up-Time
> 132370 8197
XGE0/6/1.31 GE0/11/3.31 115.1.1.2
22.22.22.22/32
579 LSP_DEFAULT 01d00h36m
> 132262 N/A
XGE0/6/1.31 inloopback 0.0.0.0
192.168.184.0/24
747 LSP_DEFAULT 01d00h36m
> 132166 N/A
XGE0/6/1.31 inloopback 0.0.0.0
192.168.88.0/24
651 LSP_DEFAULT

22

Service Configuration
Before using the show running command to query single-station configurations,
log in the device through the SecureCRT software.
IP address of the
SecureCRT-based login:
loopback interface
of the device

Both the username and password are root.

23

Service Configuration
The following page is displayed after successful login to the core switch card.

24

Service Configuration
Enter the following command and press Enter to log in to the control program.

Log in to the control

program.

25

Service Configuration
Enter the password (a) and run en, as shown in the following figure.

26

Service Configuration
The following describes the outputs of the show running-config command at
stations A1 and B1.

A1

!
no service password-encryption
Username
!
hostname A1_YZXTD
!
Password
password a
Active VC for 1X services
log file rcu.log
! mpls l2-circuit YZXTD_A11-LXRJ_B1-pw-main-1X 101 222.173.0.168 control-word
mpls l2-circuit YZXTD_A11-QLGJ_B2-pw-backup-1X 102 222.173.0.167 controlword
Standby VC for 1X services
mpls l2-circuit YZXTD_A11-LXRJ_B1-pw-main-DO 201 222.173.0.168 control-word
mpls l2-circuit YZXTD_A11-QLGJ_B2-pw-backup-DO 202 222.173.0.167 controlword

27

Service Configuration

A1

mpls class-type ct0 default

mpls class-type ct1 A11
mpls class-type ct3 a3
mpls class-type ct4 a4
mpls class-type ct5 a5
mpls class-type ct6 a6
mpls class-type ct7 a7
mpls te-class te0 default 0
mpls te-class te1 A11 1
mpls te-class te2 A11 2
mpls te-class te3 a3 3
mpls te-class te4 a4 4
mpls te-class te5 a5 5
mpls te-class te6 a6 6
Configure BFD FOR PW
mpls te-class te7 default 7

28

Service Configuration
Configured BFD for PW.

A1

mpls bfd l2-circuit 101 local-discriminator 101 remote-discriminator 101 min-tx 50 minrx 50
mpls bfd l2-circuit 102 local-discriminator 102 remote-discriminator 102 min-tx 50 minrx 50
mpls bfd l2-circuit 201 local-discriminator 201 remote-discriminator 201 min-tx 50 minrx 50
mpls bfd l2-circuit 202 local-discriminator 202 remote-discriminator 202 min-tx 50 minrx 50
Configured the VPWS.
mpls propagate-ttl
mpls vpws YZXTD_A11-LXRJ_B1-vpws-1X 101
Bound the VC to the VPWS.
vpws-vc YZXTD_A11-LXRJ_B1-pw-main-1X
vpws-vc YZXTD_A11-QLGJ_B2-pw-backup-1X
mpls vpws YZXTD_A11-LXRJ_B1-vpws-DO 201
vpws-vc YZXTD_A11-LXRJ_B1-pw-main-DO
vpws-vc YZXTD_A11-QLGJ_B2-pw-backup-DO

29

Service Configuration
Configured LDP. The IP address of the loopback

A1

router ldp
interface is configured here.
router-id 3.76.1.201
explicit-null
no-php
global-merge-capability merge-capable
Configured the IP address of the loopback
control-mode ordered
interface for oppostation end B1.
label-retention-mode liberal
pw-status-tlv
targeted-peer ipv4 222.173.0.168 Quit the TAR-PEER configuration mode.
exit-targeted-peer-mode
the IP address for
targeted-peer ipv4 222.173.0.167 Configured
loopback in the local station.
exit-targeted-peer-mode
transport-address ipv4 3.76.1.201
hold-time 15
all: LDP advertises labels for all.
hello-interval 5
host-only: LDP advertises label for hosts only.
targeted-peer-hold-time 60
for vc: LDP advertises labels for VCs only.
targeted-peer-hello-interval 20
keepalive-timeout 60 keepalive-interval 20
targeted-peer-hello-receipt
advertisement-mode downstream-unsolicited
mpls ldp lsp trigger host-only
30

Service Configuration
Configured the physical
interface.

A1

!
interface FE0/5/5
mtu-board 9600
no shutdown
switchport
Configured the subinterface of VLAN
physical basic oe-mode electric
33 in 1X services.
commit
! interface FE0/5/5.33
description for QoS-NMS
Enabled the interface.
no shutdown
Configured the interface
on the UNI side.
switchport
uni stream-type 1
Bound the VC to this interface.
uni stream-id 1 ploy 0 assign 1
uni stream-id 1 ovlan 33
mpls-l2-circuit YZXTD_A11-LXRJ_B1-pw-main-1X ethernet
mpls-l2-circuit YZXTD_A11-QLGJ_B2-pw-backup-1X ethernet secondary
vc-mode revertive
commit
31

Service Configuration
Station A1:

!
interface FE0/5/6
mtu-board 9600
no shutdown
switchport
physical basic oe-mode electric
commit
!
interface FE0/5/6.34
description for QoS-other
no shutdown
switchport
uni stream-type 1
uni stream-id 1 ovlan 34
mpls-l2-circuit YZXTD_A11-LXRJ_B1-pw-main-DO ethernet
mpls-l2-circuit YZXTD_A11-QLGJ_B2-pw-backup-DO ethernet secondary
vc-mode revertive
commit
32

Service Configuration
!
interface GE0/5/1
mtu-board 9600
no shutdown
physical advance out-limit no-shutdown
physical queue-dispatch queue-no 0 mode 1 weight 1
physical queue-dispatch queue-no 0 cir 0 pir 1000 cbs 256 pbs 256
physical queue-dispatch queue-no 1 mode 1 weight 1
physical queue-dispatch queue-no 1 cir 0 pir 1000 cbs 256 pbs 256
physical queue-dispatch queue-no 2 mode 1 weight 4
A1 physical queue-dispatch queue-no 2 cir 100 pir 1000 cbs 256 pbs 256
physical queue-dispatch queue-no 3 mode 1 weight 4
physical queue-dispatch queue-no 3 cir 100 pir 1000 cbs 256 pbs 256
physical queue-dispatch queue-no 4 mode 0 weight 0
physical queue-dispatch queue-no 4 cir 100 pir 900 cbs 256 pbs 256
physical queue-dispatch queue-no 5 mode 1 weight 4
physical queue-dispatch queue-no 5 cir 100 pir 1000 cbs 256 pbs 256
physical queue-dispatch queue-no 6 mode 0 weight 0
physical queue-dispatch queue-no 6 cir 100 pir 900 cbs 256 pbs 256
label-switching
Enable label switching.
commit
33

Service Configuration

!
Configure dthe IP address of
the subinterface.
interface GE0/5/1.31
description SD-JN-SJ-A-1.163 GE0/0/0 2.5G(S-16N0001IP)
ip address 10.1.1.5/30
Enable OSPF on this
no shutdown
interface.
label-switching
A1 commit
Enable LDP on the NNI.
ip ospf network point-to-point
ip ospf cost 100
enable-ldp ipv4 no-igp-sync

34

Service Configuration

A1

!
interface GE0/2/1
mtu-board 9600
no shutdown
physical advance out-limit no-shutdown
physical queue-dispatch queue-no 0 mode 1 weight 1
physical queue-dispatch queue-no 0 cir 0 pir 1000 cbs 256 pbs 256
physical queue-dispatch queue-no 1 mode 1 weight 1
physical queue-dispatch queue-no 1 cir 0 pir 1000 cbs 256 pbs 256
physical queue-dispatch queue-no 2 mode 1 weight 4
physical queue-dispatch queue-no 2 cir 100 pir 1000 cbs 256 pbs 256
physical queue-dispatch queue-no 3 mode 1 weight 4
physical queue-dispatch queue-no 3 cir 100 pir 1000 cbs 256 pbs 256
physical queue-dispatch queue-no 4 mode 0 weight 0
physical queue-dispatch queue-no 4 cir 100 pir 900 cbs 256 pbs 256
physical queue-dispatch queue-no 5 mode 1 weight 4
physical queue-dispatch queue-no 5 cir 100 pir 1000 cbs 256 pbs 256
physical queue-dispatch queue-no 6 mode 0 weight 0
physical queue-dispatch queue-no 6 cir 100 pir 900 cbs 256 pbs 256
label-switching
commit
35

Service Configuration

A1

!
interface GE0/2/1.31
description SD-JN-SJ-A-1.163 GE0/0/0 2.5G(S-16N0001IP)
ip address 10.1.1.1/30
no shutdown
label-switching
commit
ip ospf network point-to-point
ip ospf cost 100
enable-ldp ipv4 no-igp-sync
!

36

Service Configuration
Configured OSPF.
!
router ospf 2
ospf router-id 3.76.1.201
timers spf exp 50 50000
area 0.0.0.1 stub
network 3.76.1.201/32 area 0.0.0.1
network 10.1.1.0/30 area 0.0.0.1
network 10.1.1.4/30 area 0.0.0.1
cspf disable-better-protection
A1 !
telnet max-clients 10
line vty
exec-timeout 0 0
login
!
dhcp enable
dhcp hops threshold 16
!
end

37

Set the IP address of the OSPF

interface to the IP address of the
loopback interface.
Added the loopback
interface to area 1.
Added the link interface to area 1.

CLI basic commands in total

38

View global configuration

CLI : show running
Screenshot below: (CLI in the red box)

39

View all ARP

CLI : show arp

Screenshot below: (CLI and Content displayed as red font)

40

View port information

CLI : show ip interface brief

Screenshot below: (CLI and Content displayed as red font)

41

View the IGP route table

CLI : show ip route
Screenshot below: (CLI and Content displayed as red font)

42

View RSVP session information

CLI : show rsvp session

Screenshot below: (CLI and Content displayed as red font)

43

View RSVP enabled interface information

CLI : show rsvp interface
Screenshot below: (CLI and Content displayed as red font)

44

View all LSP initiated by this router

CLI : show mpls forwarding-table

Screenshot below: (CLI and Content displayed as red font)

45

View all mpls ilm table information

CLI : show mpls ilm-table
Screenshot below: (CLI and Content displayed as red font)

46

View bfd session information

CLI : show bfd session
Screenshot below: (CLI and Content displayed as red font)

47

View bfd enabled interface

CLI : show bfd interface
Screenshot below: (CLI and Content displayed as red font)

48

View LDP session information

CLI : show ldp session
Screenshot below: (CLI and Content displayed as red font)

49

View LDP enabled interface

information
CLI : show ldp interface

Screenshot below: (CLI and Content displayed as red font)

50

View all VC FEC statistics

CLI : show mpls vc-table
Screenshot below: (CLI and Content displayed as red font)

51

Contents

1.

Checking Services using OTNM2000 GUI

2.

Checking Services using Command Line

Interface (CLI)

3.

Fault Isolation based on different scenario

52

Fault Isolation
E-CES(E1) Service
Situation : Field Operations is conducting loop testing end-to-end(RNC to end site) but they
cant see the loop provided or the data cant pass thru RNC from the Source Site.
Checking:
1. Check the slot/port of the Source Site and Destination Site if mapping is created on
Business Management View.
2. Check the NE E1K5/E101/S101/S102 Card for current alarms. If seen PPI_LOS alarm,
physical termination fault. If seen AIS alarm, possible that alarm is on far end site.
3. If no alarm seen, check the port if unshut. Go to Secure CRT, type show run int <port>
command or go to STATUS>Link State must be with on status.
4. If port is up, check the status of VC by running the command show mpls vc-table and
check its uptime status. If with INACTIVE status, check the status of the tunnel by running
the command show rsvp session.
5. If no result show upon running the mpls vc-table command, check the LDP session if
configured by running show ldp session.
6. If RSVP session is down, check for active fiber cut.
6. If all protocols are up, conduct loop testing with the Engineer on site.

53

Checking Services using OTNM2000 GUI

Control Command

E1 Loop Testing: Verify the configured E1 ports on Business Management .

SITE A

SITE B

Accessing C31 Ilagan on FH NMS:

LLB: Local Loopback = enable loop at Site A DDF
facing the BER tester(also installed in Site A)

DLB: Device Loopback = enable loop at Site A

facing the DDF of Site B(depending on the created
tunnel/trail, where BER tester is installed in Site B).

54

Fault Isolation
E-LAN(Data Services (3G, BRO, Wimax, FD/TD-LTE)
Down Service
Checking:
1. Check the slot/port of the Source Site and Destination Site if mapping is
created on Business Management View.
2. Check the port for current alarms. If seen RLOS alarm, possible physical cable
fault. If seen LINK_LOS alarm, possible port setting problem(duplex/speed).
3. If no alarm seen, check the port if unshut. For R865, select the card, view the
STATUS>>Port Status to check for link status, received/transmit packets, current
speed/duplex mode.
4. If port is up but no packets, check the status of VC by running the command
show mpls vc-table and check its uptime status. If with INACTIVE status, check
the status of the tunnel by running the command show rsvp session.
5. If no result show upon running the mpls vc-table command, check the LDP
session if configured by running show ldp session.
6. If RSVP session is down, check for active fiber cut.
7. If no fiber cut, check for broken tunnel by running show run rsvp-path.
55

Fault Isolation
L2 services are unavailable.
1) Run show mpls vc-table to check whether the VC is normal.

Vc-name: VC name
Vc-id: VC ID
Network-intf: network interface
In label: incoming label
Tunnel-label: label of the bound tunnel
Nexthop: IP address of the next-hop interface
56

Access-intf: bound interface

Out label: outgoing label
Status: status
Up-time: establishment time

Fault Isolation
The VC is abnormal.

57

Fault Isolation
2) If the VC is abnormal, run show mpls forwarding-table to check the LSP
labels generated by LDP. The outgoing label is normal.

58

Fault Isolation
The outgoing label is abnormal.

59

Fault Isolation
Run show mpls ilm-table to check the incoming label. The incoming label is normal.

60

Fault Isolation
The incoming label is abnormal.

61

Fault Isolation
3) If the LDP connection is abnormal, run show ip ospf neighbor to check OSPF.

Neighbor id: IP address of the peer loopback interface Pri: priority State: status
Dead time: The connection is interrupted if no Hello packet is received from the peer end within 30
seconds.
Address: IP address of the peer interface
Interface: local interface
Instance id: local OSPF process ID (0 by default) Up time: establishment time
62

Fault Isolation
The OSPF connection is abnormal.

63

Fault Isolation
If the OSPF connection is abnormal, ping the directly-connected interface. If the
directly-connected interface can be successfully pinged, OSPF is faulty.

64

Fault Isolation
The directly-connected interface cannot be pinged.

5) If the directly-connected interface cannot be pinged, check the link state to

see whether the optical fiber is correctly connected.
65

THANK YOU

66