CS XGSN TS V81 08 10

GPRS/UMTS Troubleshooting
Technical Training
08-10-2010
Version Number 8.1
Americas Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: 408 526-4000
800 553-NETS (6387)
Fax: 408 527-0883
Cisco Confidential and Proprietary

Do Not Distribute
THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS,
INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED
WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED
WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.
The following information is for FCC compliance of Class A devices: This equipment has been tested and found to comply with the limits for a Class A digital device, pursuant to part 15
of the FCC rules. These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment. This equipment
generates, uses, and can radiate radio-frequency energy and, if not installed and used in accordance with the instruction manual, may cause harmful interference to radio communications.
Operation of this equipment in a residential area is likely to cause harmful interference, in which case users will be required to correct the interference at their own expense.
The following information is for FCC compliance of Class B devices: This equipment has been tested and found to comply with the limits for a Class B digital device, pursuant to part 15
of the FCC rules. These limits are designed to provide reasonable protection against harmful interference in a residential installation. This equipment generates, uses and can radiate radio
frequency energy and, if not installed and used in accordance with the instructions, may cause harmful interference to radio communications. However, there is no guarantee that
interference will not occur in a particular installation. If the equipment causes interference to radio or television reception, which can be determined by turning the equipment off and on,
users are encouraged to try to correct the interference by using one or more of the following measures:
Reorient or relocate the receiving antenna.
Increase the separation between the equipment and receiver.
Connect the equipment into an outlet on a circuit different from that to which the receiver is connected.
Consult the dealer or an experienced radio/TV technician for help.
Modifications to this product not authorized by Cisco could void the FCC approval and negate your authority to operate the product.
The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCBs public domain version
of the UNIX operating system. All rights reserved. Copyright 1981, Regents of the University of California.
NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED AS IS WITH ALL
FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE
PRACTICE.
IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT
LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS
HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
Cisco and the Cisco Logo are trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and other countries. A listing of Cisco's trademarks can be found at
www.cisco.com/go/trademarks. [Insert any required attribution of a third party mark.] [Other] Third party trademarks mentioned are the property of their respective owners. The use of
the word partner does not imply a partnership relationship between Cisco and any other company.
Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output,
network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is
unintentional and coincidental.
Gprs/Umts Trouleshooting
2010 Cisco Systems, Inc. and/or its affiliated entities. All rights reserved.

Do Not Distribute
Table of Contents
Lesson 1: ST16/ASR5000 Hardware Troubleshooting
1-1
Lesson 2: ST16/ASR5000 Software Troubleshooting
2-1
Lesson 3: ST16/ASR5000 Logging
3-1
Lesson 4: ST16/ASR5000 Alarms and Traps
4-1
Lab: Troubleshooting the System
Lesson 5: ST16/ASR5000 Troubleshooting Exercise

Lab: Debugging a GGSN on ASR5000

Do Not Distribute
4-15
5-1
5-5

Do Not Distribute
GPRS/UMTS Trouleshooting
ST16/ASR5000
Hardware
Troubleshooting
Lesson 1
2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Do Not Distribute
1-1
Lesson Overview
This module outlines a methodology for troubleshooting the hardware of the
ST16/ASR5000.
Starting with a broad view, important CLI commands are presented and illustrated
that can be used to verify the integrity of the system. Then you will be shown CLI
commands that allow you to take a close look at each card and each port.
1-2

Do Not Distribute
Lesson Overview
In this module, a hardware troubleshooting
methodology will be presented.
Hardware-related CLI commands will be introduced
that provide detailed information about the system,
each card, and each port

Do Not Distribute
1-3
Chassis General View

The slide on the opposite page shows a high-level view of basic chassis
components, for either the ST16 or ASR5000.
The most useful CLI commands for viewing the fan and power subsystems are:
show fans
show power
show power all
show led all
show led <slot>
show temp
show maximum-temperatures
The show temp command displays the present temperature of each installed card.
The show maximum-temperatures command shows the highest temperature
reached on each card, and when it occurred.
1-4

Do Not Distribute
Chassis General View

Useful CLI commands:
show fans
Mounting brackets
show power
show power all
Upper fan tray
show led all

show led <slot>
show temp
Application
cards
Line cards
Fan tray bezels
Lower Fan Tray &

Particulate Air Filter
Power filter units

Do Not Distribute
1-5
Viewing Card-level Details

The following CLI commands listed on the opposite page are most useful for viewing
details of each card and the relationships between the cards.
The table below provides detail on some acronyms that appear:
Applicable
Card(s)
Field
Description
SPC/SMC,
PAC/PSC
Which Switch Fabric modes the card can do. Possible values are:
- control plane
- switch fabric
Card
Programmables
All
This field indicates if the software on any of the programmable

components on the card is not at the current revision.
If all software is current, Up To Date is displayed. If one or more
components do not have the most current software, Out of Date is
displayed along with the component(s) requiring the newer code.
Slave SCB
PAC/PSC, SPIO,
RCC, and line
cards
The firmware version of the component that allows non-SPC/SMC cards

to communicate with the SPC/SMC over the system control bus (SCB).
on-card The version of the firmware that on the boot flash for the
component.
SPC CPLD
SPC
The version number of one of the Complex Programmable Logic

Devices (CPLD)s that allow the SPC to communicate with PACs.
SRM
SMC
The firmware version of the Status, Reset, and Monitoring component.
CIF FPGA
SMC
The firmware version of the Chassis Information (CIF) Field

Programmable Gate Array (FPGA).
on-card indicates the version of the firmware that on the boot flash for
the component. running is the firmware version that is currently
operational.
PSR
PSC
The firmware version of the Power, Status, and Reset component.
BIOS
PSC, SMC
The version of the Basic Input/Output System (BIOS).

the component.
DT FPGA
PSC
The firmware version of the Data Transport (DT) Field Programmable

Gate Array (FPGA).
the component. running is the firmware version that is currently
operational.
SFP Info (Port

1/2)
Indicates information pertaining to the small form-factor pluggable (SFP)

SPIO and
modules installed on the card. Among the information provided is the
Ethernet 1000 line
Manufacturer name, ID number, the modules part and serial numbers,
cards
and the production date.
Switch Fabric
Modes
1-6

Do Not Distribute
Viewing Card-level Details

At the card-level, these commands are
useful:
show hardware inventory
show hardware card <slot #>
show card table
show card mappings
show card info <slot #>
show card diag <slot #>
Slot 25
SPIO (standby
Slot 32
Slot 24
SPIO (active)
S S
P
P
I
O I
(R)O
Slot 17
RR
CC
CC
Slot 48
Slot 33
Slot 1
Chassis rear
Slot 41
RCC
Slot 40
RCC
Slot 16
Slot 8
SPC/SMC (active)
Chassis front
Slot 9
SPC/SMC (standby)

Do Not Distribute
1-7
Viewing Port-level Status Detail

The commands displayed on the opposite page are most useful for obtaining status
information on any LAN port of the system.
The table below provides some details of important fields in the output:
Field
Description
Administrative
State
Indicates whether or not the card has been configured for use via software. If it has
been configured, Enabled will be displayed. If not, Disabled will be displayed.
Description
The description given to the port during software configuration. If no description

was configured, (None Set) will be displayed.
MAC Address
The media access control (MAC) address for the port. If Virtual MAC addressing is
enabled the MAC address displayed is followed by (Virtual).
Link State
The link status, either Up or Down.
Link Duplex
The actual duplex mode currently being used for the link. Either Full or Half will be
displayed.
Link Speed
The actual data rate currently being supported by the port. Either 10 Mb, 100 Mb,
or 1000 Mb will be displayed.
Operational
State
The operational state and mode of the card. The operational state is listed first
followed by a comma ( , ), then the operational mode. The operational state is
listed as either Up or Down. The operational mode of the card that the port belongs
to. The card can be in one of the following modes:
Active: Indicates that the card is an active component that will be used to process
subscriber data sessions.
Standby: Indicates that the card is a redundant component. Redundant
components will become active through manual configuration or automatically
should a failure occur.
Offline: Indicates that the card is installed but is not ready to process subscriber
data sessions. This could be due to the fact that it is not completely installed (i.e.
the card interlock switch is not locked, refer to the System Installation Guide for
information on installing cards in the system) or that its processes have been
halted.
1-8

Do Not Distribute
Viewing Port-level Status Detail

Useful CLI commands for
viewing details any LAN
interfaces:
show port table
show port info <slot #>
Management
LAN Interfaces
Console Port
GE Lan
Interface

Do Not Distribute
1-9
Viewing Port-level Counters

The CLI commands listed on the opposite page can be used to look at counters on
any LAN port of the system.
The table below provides some detail on certain fields of the output:
Field
Applicable
Line Card(s)
Description
RX OVF
FELC/GELC
The number of overflows received.
RX
SHORT
OK
FELC/GELC
The number of frames, less than 64 bytes in length,

received without any error.
RX
SHORT
CRC
FELC/GELC
The number of frames, less than 64 bytes in length,

received with cyclical redundancy check (CRC) error.
RX
NORM
CRC
FELC/GELC
The number of frames, with lengths between 64 bytes

and the maximum frame size, received with an integral
number of bytes and a cyclical redundancy check
(CRC) error.
RX
LONG
OK
FELC/GELC
The number of frames, larger than the maximum frame

size, received without any error.
RX
LONG
CRC
FELC/GELC
The number of frames, larger than the maximum frame

size, received with CRC error.
RX
PAUSE
FELC/GELC
The number of correct received flow-control frames.
RX FALS FELC/GELC
CRS
The number of false carrier events detected.
RX
GPCS
ERR
GELC
The number of received frames during which physical

(PHY) symbol errors were detected.
TX
PAUSE
FELC/GELC
The number of correct transmitted flow-control frames.
TX ERR
FELC/GELC
The number of frames transmitted with an error due to

transmit FIFO underflow or TXERR signal assertion
1-10

Do Not Distribute
Viewing Port-level Counters

Useful CLI commands for viewing traffic counters:
show port datalink counters <slot/port>
clear port datalink counters <slot/port>
show port utilization table
show port npu counters <slot/port> vlan
<vlanid>
clear port npu counters

Do Not Distribute
11
1-11
1-12

Do Not Distribute
12
ST16/ASR5000 Software
Troubleshooting
Lesson 2

Do Not Distribute
2-1
Lesson Overview
This module provides a collection of CLI commands that could be used for
troubleshooting the software of the ST16/ASR5000.
The CLI commands shown will allow you to:
2-2
check the overall health of the operating system.
gain some visibility into the call processing behavior of the system.

Do Not Distribute
Lesson Overview
Software-related CLI commands will be introduced
that provide:
Detailed information about the operating system
Insight into call processing behavior

Do Not Distribute
2-3
Task Health Check

There are many Control Processors (CPs), or CPUs, on the ST6/ASR5000. On
each of these CPs, many tasks are started. Some of these tasks are directly related
to call signaling, and some are there in support of call signaling.
The best way to verify the overall health of all the system tasks is to look at the task
resources with the following command:
show task resource
This command displays:
2-4
Each task that is running
On what CP it is running
Allocated/used CPU time
Allocated/used memory
Allocated/used files
Allocated/used session
General Status

Do Not Distribute
Task Health Check

Use this command to check the
overall health of software tasks:
(active)
(standby)
SMC
show task resource
This command provides more

detail on hierarchy of task
creation:
SMC
vpnctrl
Controller
tasks
sessctrl
Control Bus
show task table
CP0
CP0
aaamgr
sessmgr
sessmgr
CP0
aaamgr
vpnmgr
a11mgr
CP1
CP1
sessmgr
aaamgr
PSC
CP1
aaamgr
sessmgr
PSC

Do Not Distribute
PSC tasks
sub-manager
2-5
Detailed Task Information

Using the CLI command shown on the opposite page, you can view details of
individual tasks.
It is useful to do a show task info on the session manager facility. The output of this
command will show you:
how many sessions are being handled by the manager, and from what
service.
the maximum number of sessions, per service, that have been handled by
this instance of the session manager.
Below is the full output for one session manager instance:

[local]training# show task info facility sessmgr all | more
Task sessmgr instance 28:
Location
: Card 4, CPU 0, Pid 824
Parent
: Task sessctrl instance 0 (Card 8, CPU 0, Pid 916)
CPU usage
: 0.0% used, 100.0% allocated, 0.6% max
File Usage
: 26 used, 500 allocated, 44 max
Memory usage
: 105.9MB used, 600.0MB allocated, 106.3MB max
PDSN sessions
: 0 used, 3 max
HA sessions
: 0 used, 2 max
GGSN sessions
: 0 used, 0 max
L2TP LNS sessions : 0 used, 0 max
Combo Phone sessions : 0 used, 0 max
ECSv2 sessions
: 0 used, 0 max
IP Services Gateway : 0 used, 0 max
EV-DO Rev A PDSN
: 0 used, 0 max
SCM Proxy/Registrar IETF RFC3261 : 0 used, 0 max
SCM Proxy P-CSCF
: 0 used, 0 max
SCM Serving S-CSCF : 0 used, 0 max
SCM Interrogating I-CSCF : 0 used, 0 max
ASN GW
: 0 used, 0 max
Total sessions
: 0 used, 26400 allocated, 4 max
Status
: In-Use
2-6

Do Not Distribute
Detailed Task Information

To view details of a specific
task, use this command:
show task info
facility
<facility_name>
instance <id>
(active)
(standby)
SMC
SMC
vpnctrl
sessctrl
This is good to see what each

session manager is doing:
show task info
facility sessmgr all
Control Bus
CP0
CP0
aaamgr
sessmgr
sessmgr
CP0
aaamgr
vpnmgr
a11mgr
CP1
CP1
sessmgr
aaamgr
PSC
CP1
aaamgr
sessmgr
PSC

Do Not Distribute
PSC
2-7
Licensed Sessions Versus Actual Sessions

Sometimes it is necessary to see how close to the licensing limits the system is
operating. This can be done with the following command:
show resource session
The following information will be displayed:
Number of session managers on the system
Number of session managers that are no longer accepting calls
Number of standby sessions managers
For every licensed service, shows

Number of active sessions
maximum number of active session
license limit
2-8

Do Not Distribute
Licensed versus Actual Sessions

This command displays the number of active sessions per
service (and license limit):
[local]training# show resources session | more
Session Information:
In-Use Session Managers:
Number of Managers : 30
Capacity
: 198000 min / 396000 typical / 792000 max
Usage
: 0
Busy-Out Session Managers:
Capacity
: 0 min / 0 typical / 0 max
Usage
: 0
Standby Session Managers:
PDSN Service:
Actual
In Use
: 0
Max Used
: 10
Limit
: 10000
License Status
: Within Acceptable Limits
.
.

Do Not Distribute
Licensed
2-9
SGSN Interaction with BSC/RNC

The commands shown on the opposite page provide some details on the
ST16/ASR5000s communication with remote BSCs/RNCs.
Sometimes it is useful to obtain a list of all RNCs that have communicated with the
SGSN. This type of list can be obtained by using the:
show ss7-routing-domain x routes command, for the SS7 routing domain that is
related to the IuPS / RNC communication side. This will enable you to ascertain
connectivity to the RNCs .
Another useful command is the show sccp-network 1 status all command that will
show you if the subsystem (142) on the DPC you suspect as problematic shows up
as available
The show session setuptime sgsn-only command lists ranges of time, in one
hundred millisecond increments and one second increments. It then shows a count
of existing sessions and where in these ranges they fell in terms of call setup time.
The show session duration sgsn-only command also provides a count of inprogress calls and their duration.
The show session disconnect-reasons sgsn-only provides a list of disconnect
reasons with a count associated with each reason, as well as the percentage of total
calls that count represents.
2-10

Do Not Distribute
SGSN Interaction with RNC

Use the following command to view a list of RNCs with which that
SGSN is communicating:
show ss7-routing-domain x routes
show sccp-network x status all
Use the following commands to view information about the

SGSNs relationship with RNCs:
show session setuptime sgsn-only
show session duration sgsn-only
show session disconnect-reasons
RNC
SGSN
RNC

Do Not Distribute
11
2-11
SGSN GMM Statistics

The commands shown on the opposite page provide some details on the GMM
functionality of the SGSN. This provides multiple counters here is a description of
just a few of them:
Field
Description
Attached Subscribers
Statistics for attached subscribers.
Total Attached
Total subscribers attached for 2G and 3G.
3G Attached
Total subscribers attached for 3G only.
2G Attached
Total subscribers attached for 3G only.
Home Subscribers
Statistics for attached home subscribers.
Total Home
Total home subscribers attached for 2G and 3G.
3G Home
Total home subscribers attached for 3G only.
2G Home
Total home subscribers attached for 3G only.
Visiting National Subscribers
Statistics for attached visiting national subscribers.
Visiting Foreign Subscribers
Statistics for attached visiting foreign subscribers.
Total-Visiting-Foreign
Total visiting foreign subscribers attached for 2G and 3G.
3G-Visiting-Foreign
Total visiting foreign subscribers attached for 3G only.
2G-Visiting-Foreign
Total visiting foreign subscribers attached for 2G only.
Activated Subscribers
Indicates the statistics of activated subscribers.
Total Activated
Total number of activated 2G and 3G subscriber.
3G Activated
Total number of activated 3G subscribers only.
2G Activated
Total number of activated 2G subscribers only.
Activate PDP Contexts
Indicates the statistics of activated PDP contexts.
Total Actv PDP Ctx
Total number of activated 2G and 3G PDP contexts.
3G-Actv Pdp Ctx
Total number of activated 3G PDP contexts only.
2G-Actv Pdp Ctx
Total number of activated 2G PDP contexts only.
Total Actv Pdp Ctx with Direct

Tunnel
Total number of activated PDP contexts through direct tunnel.
Message Statistics
Indicates the statistics of messages.
IMSI
Indicates the statistics of total attach through international mobile subscriber

identity (IMSI).
Total-IMSI-Attach
Total international mobile subscriber identity (IMSI) attach including 2G and

3G.
2-12

Do Not Distribute
SGSN GMM statistics

Use the following command to view mobility statistics
for the SGSN / RNC interactions:
show gmm-sm statistics { gmm-only [ gprsservice srvc_name ] | gprs-service srvc_name |
sgsn-service srvc_name [ rnc mcc mc_code mnc
mn_code rnc-id rnc_id | routing area mcc
mc_code mnc mn_code lac loc_code rac ra_code ]
| sm-only [ gprs-service srvc_name ] } [
verbose ] [ | { grep grep_options | more }]
ss7-routing-domain x routes

Do Not Distribute
13
2-13
MAP Details
You can get multiple counters related to the MAP operations of the SGSN using the
show map statistics.
Here is an example of the output of this command:
MAP Statistics:
Open Req TX:
Open Response TX:
Close Req TX:
Abort Req TX:
Authentication Req TX:
Failed:
Check IMEI Req TX:
Failed:
GPRS Loc Upd Req TX:
Failed :
Ins Sub Data(ISD)Reqs RX:
ISD Rsp TX :
Del Sub Data Req RX :
Failure Rsp TX :
Auth Fail Rep Req TX :
Failed :
Cancel Location Req RX :
Failure Rsp TX :
Purge Req TX :
Failed :
MO FWD Req TX :
Failed :
MT FWD Req RX :
Failure Rsp Tx:
Ready For SM Req TX :
Failed :
HLR RESET RX :
General Stats:
Open Rsp Failure :
Version Mismatch :
2-14
2
Open Req RX:
0
Open Rsp RX:
2
0
Close Req RX:
2
0
Abort Req RX:
0
4
Successful:
4
0
Timed Out
0
Successful:
0
0
Timed Out
1
Successful:
1
0
Timed Out:
1
StandAlone ISD Req RX:
0
1
ISD Failure Rsp TX:
0
Successful Rsp TX:
0
0
0
Successful:
0
0
Timed Out
0
Successful Rsp TX:
0
0
0
Successful:
0
0
Timed Out:
0
Successful:
0
Timed Out:
0
Successful Rsp Tx:
0
0
Successful:
0
Timed Out:
0
0
0
0

Do Not Distribute
0
0
0
0
0
0
0
0
0
0
MAP protocol Details (SGSN)

To obtain the MAP counters for a particular service:
show map statistics [name <service_name>]
To verify map connectivity to HLR / EIR etc.:

show sccp-network x status all

Do Not Distribute
15
2-15
GTP-related Statistics
The GTP tunneling operation of the system can be observed using one of the
commands shown on the opposite page. You can view GTP statistics per SGSN,
Per GGSN, per APN etc.
You can also view GTP statistics per subscriber as long as you know either the
callid, username, msid, or imsi. This, too, is shown on the opposite page.
2-16

Do Not Distribute
GTP related Statistics

GTP behavior can be monitored on SGSN or GGSN Service:
SGSN service show sgtp statistics { all | gsn-address | sgtp-service | verbose }
GGSN Service show gtpc { all | APN | ggsn-service | sgsn-address | . }. The Same
applies for GTP-U.
GTP counters can also be obtained per subscriber:

show gtpc counters { all | callid call_id | imsi id | msid ms_id | username user_name }
GTP Stats per

- GGSN service
- SGSN service
- subscriber
HLR
VLR
MSC
SS7
Network
Radius
AAA
RAN
Iu
Interface
BTS
BSC
Mobile
Node
Gn
Core Network
Gn
Gi
RNC
PDN
Radio Tower
SGSN
GGSN

Do Not Distribute
17
2-17
RADIUS Commands
A detailed look at the RADIUS subsystem on the ST16/ASR5000 can be obtained
using the commands shown on the opposite page.
However, it is important to remember that these commands execute within a
context, and if there are no RADIUS servers in that context you will not receive any
output. So you must first change your context to that which contains some RADIUS
configuration, and then execute the commands.
2-18

Do Not Distribute
RADIUS Commands (GGSN)

To obtain a summary of the RADIUS server(s) status:
show radius authentication servers detail
To obtain some statistics about RADIUS operation:

show radius counters server <ip of server>
HLR
VLR
MSC
SS7
Network
Radius
AAA
RAN
Iu
Interface
BTS
BSC
Mobile
Node
Gn
Core Network
Gn
Gi
RNC
PDN
Radio Tower
SGSN
GGSN

Do Not Distribute
19
2-19
Looking at service status

The commands shown on the opposite page are very similar, and will allow us to
verify that a specific service is in a started state and relevant parameters.
2-20

Do Not Distribute
Looking at Service Status

Commands to show the current status of system
services:
show
show
show
show
show
show
ggsn-service all|name <service_name>

sgsn-service all|name <service_name>
gprs-service all|name <service_name>
map-service all|name <service_name>
sgtp-service all|name <service_name>
iups-service all|name <service_name>
HLR
VLR
SS7
Network
MSC
Home
AAA
AAA
RAN
Mobile
Node
R-P
Interface
BTS
BSC
Pi
IP Network
Pi
PDN
PCF/RNC
Correspondent
Node (CN)
Radio Tower
PDSN serving
as Foreign
Agent (FA)
Home Agent
(HA)

Do Not Distribute
21
2-21
2-22

Do Not Distribute
22
ST16/ASR5000
Logging
Lesson 3

Do Not Distribute
3-1
Lesson Overview
This module introduces and defines the various types of logging that can be done
on the system. System logs are generated by various facilities on the system. You
can control the verbosity of these logs as well as which facility adds information to
the system log.
There are five types of logging possible: runtime, active, trace, monitor, and crash.
All of these are discussed in this lesson.
3-2

Do Not Distribute
Logging Types on the System

There are five types of logging in the system
Runtime
Active
Trace
Monitor
Crash
These are discussed in more detail on the following

slides

Do Not Distribute
3-3
Runtime Logging
Event logs are stored in a memory buffer on the active management module. They
are not stored on the flash card.
The buffer used for event log storage can store up to 50,000 events. Once this
buffers reaches its capacity, the oldest information is removed to make room for the
newest. To prevent the loss of log data, the system can be configured to transmit
logs to a syslog server over a network interface.
By default, all facilities on the system are configured to submit events to the event
log that are of a severity level error. This is referred to as runtime logging.
To see a list of facilities that can contribute to the event log, and the severity level
they are set to, you can use the following command:
show logging
To view the contents of the system log (memory buffer), use the following command:
show logs
3-4

Do Not Distribute
Runtime Logging
These are logs that are generated by various facilities on the system
Enabled by default at system boot, set a a severity level of error
Output sent to SPIO console and stored in active memory buffer on
SMC/SPC module
Buffer has 50,000 event capacity
When full, wraps around to beginning (oldest events lost)
Optionally, output can be sent to an external syslog server
Use show logging command to view default runtime logging settings
System facilities
CLI: show logging
aaaclient
aaamgr
aaaproxy
.
.
user-data
user-l3tunnel
vpn
Active Buffer
(to syslog server)
Capacity: 50,000K events
CLI: show logs

Do Not Distribute
3-5
Configuring Runtime Logging

You an configure a logging filter that determines which system facilities should be
logged, and at what levels. By default, all facilities are configured with a logging filter
that is set to an error level. The CLI command you use is the following:
logging filter runtime facility facility level report_level
[ critical-info | no-critical-info ]
The table below provides more information about the variables of this command:
Keyword/Variable
Description
facility
Specifies the facility to log information for. A facility is a protocol or task that is
in use by the system. a complete list of facilities that can be seen with the
show logging command.
report_level
Specifies the log severity level for the filter. report_level can be one of the
following severity level:
critical: Logs only those events indicating a serious error has occurred that
is causing the system tor a system component to cease functioning. This is
the highest severity level.
error: Logs events that indicate an error has occurred that is causing the
system or a system component to operate in a degraded state. This level also
logs events with a higher severity level.
warning: Logs events that may indicate a potential problem. This level also
unusual: Logs events that are very unusual and may need to be
investigated. This level also logs events with a higher severity level.
info: Logs informational events and events with a higher severity level.n
trace: Logs events useful for tracing and events with a higher severity level.
debug: Logs all events regardless of the severity.
NOTE: It is recommended that a level of error or critical be configured to
maximize the value of the logged information while minimizing the quantity of
logs generated.
critical-info
Specifies that events with a category attribute of critical information are to be

displayed. Examples of these types of events can be seen at bootup when
system processes and tasks are being initiated.
no-criticalinfo
Specifies that events with a category attribute of critical information are not to
be displayed.
3-6

Do Not Distribute
Configuring Runtime Logging

The level of detail provided by the various ST facilities can be changed
from the default of error
Use the following commands to configure runtime logging filters
config
logging filter runtime facility <facility name>
level <report-level> critical
Changes level of
detail in buffer
System facilities
aaaclient
aaamgr
aaaproxy
.
.
user-data
user-l3tunnel
vpn
Active Buffer
(to syslog server)

Do Not Distribute
3-7
Runtime Logging syslog server

It is good practice to have the contents of the system log sent to a remote syslog
server, greatly decreasing the chances of losing valuable information.
Information generated by the run time facilities can be transmitted to a syslog server
for permanent storage. The command to do this is:
logging syslog ip_address [ facility facilities ] [ rate value
] [ pdu-verbosity pdu_level ] [ pdu-data format ] [ eventverbosity event_level ]
The table below provides some details:
Keyword/Variable
syslog ip_address
Description
Specifies the IP address of the Syslog server. ip_address is an IP
address in dotted decimal notation.
Specifies the local facility for which the logging options shall be applied
as one of facilities:
facility facilities
local0, local1, local2, local3, local4, local5, local6, or local7.The default
is local7.
rate value
Specifies the rate in which log entries are allowed to be sent to the
system log server. value must be an integer from 0 through 100000.
The default is 1000 events per second.
pdu-verbosity
pdu_level
Specifies the level of verboseness to use in logging of protocol data units

(PDUs) as a pdu_level value from 1 to 5 where 5 is the most detailed.
This affects only protocol logs.
pdu-data format
Specifies output format for protocol data units when logged as one of
format:
none: Displays results in raw, or unformatted text
hex: Displays results hexadecimal format
hex-ascii: Displays results in hexadecimal and ASCII similar to a mainframe dump
event-verbosity
event_level
Specifies the level of detail to use in logging of events as one of

event_level:
min: Provides minimal information about the event such as event
name, facility, event ID, severity level, date, and time.
concise: Provides detailed information about the event, but does not
provide the event source.
full: Provides detailed information about the event including the source
information identifying the task or subsystem that generated the event.
3-8

Do Not Distribute
Runtime Logging Syslog Server

Configuration of a syslog server should be done in local
context
Use this command to configure a syslog server to which
you want to send the runtime log messages
logging syslog ip_address facility local7 rate
1000 pdu-verbosity 2 pdu-data hex-ascii eventverbosity concise
Changes need to be saved to config file

System facilities
aaaclient
aaamgr
aaaproxy
Buffer
(to syslog server)

Do Not Distribute
3-9
Active Logging
Active logging is when the system is configured to send events to a specific CLI
instance. Active logging is configurable on a CLI instance-by-CLI instance.
Each active log can be configured with filter and display properties that are
independent of those configured globally for the system. Active logs are displayed in
real time as they are generated and are not written to the active memory buffer by
default.
The command to configure active logging for a particular CLI instance is:
logging active
logging filter active facility facility level report_level [
critical-info | no-critical-info ]
Keyword/Variable
Description
facility
Specifies the facility to log information for. A facility is a protocol or task that is
in use by the system. a complete list of facilities that can be seen with the
show logging command.
report_level
following severity level:
critical: Logs only those events indicating a serious error has occurred that
is causing the system tor a system component to cease functioning. This is
the highest severity level.
error: Logs events that indicate an error has occurred that is causing the
system or a system component to operate in a degraded state. This level also
warning: Logs events that may indicate a potential problem. This level also
unusual: Logs events that are very unusual and may need to be
investigated. This level also logs events with a higher severity level.
info: Logs informational events and events with a higher severity level.n
trace: Logs events useful for tracing and events with a higher severity level.
debug: Logs all events regardless of the severity.
NOTE: It is recommended that a level of error or critical be configured to
maximize the value of the logged information while minimizing the quantity of
logs generated.
critical-info
Specifies that events with a category attribute of critical information are to be

displayed. Examples of these types of events can be seen at bootup when
system processes and tasks are being initiated.
no-criticalinfo
Specifies that events with a category attribute of critical information are not to
be displayed.
3-10

Do Not Distribute
Active Logging
Displayed in real-time within a CLI-user instance
Allows a CLI user (with operator priv or above) to view events of specific
facilities
CLI users can independently set their own logging display and filtering
properties
Logging output sent to CLI display (not stored in buffer on SPC/SMC

module)
Does affect system performance
Works like this:

1.
Turn on active logging: logging active
2.
Select the facility/error_level: logging filter active facility

<facility> level <level>
3.
Turn off active logging: no logging active (or exit from CLI
session)
CLI user 1
(logging active)
display
facility
info

Do Not Distribute
11
3-11
Trace Logging
Trace logging can be used to quickly isolate issues that may arise for a specific
subscriber session that is already connected. Traces can be started for a particular
session based on call identification (callid) number, IP address, mobile station
identification (MSID) number, or username.
Note that the session must be up before it can be traced. This type of logging
cannot be used to view logging information about a call that is being established, or
about to be established.
Trace logging is a refined form of active logging: from an active trace of the session
manager, you are filtering all output except that of a single subscriber (session).
The output from trace logging goes directly into the system log. There is nothing
displayed in the CLI session where the trace was started.
Also note that trace logging can affect performance. Under heavy system load, they
can be intrusive to the processing of a session.
The command used to start a trace is shown in the slide. The table below provides
some details:
Keyword/Variable
Description
callid call_id
Indicates a trace log will be generated for a session identified by the call
identification number. call_id is specified as a 4-byte hexadecimal number.
ipaddr
ip_address
Indicates a trace log will be generated for a session identified by the IP

address. ip_address is expressed in dotted decimal notation.
msid ms_id
Indicates a trace log will be generated for a session identified by the mobile
station identification (MSID) number. ms_id must be from 7 to 16 digits
specified as an IMSI, MIN, or RMI.
name username
Indicates a trace log will be generated for a session identified by the

username. username is the name of subscriber that was previously
configured.
3-12

Do Not Distribute
Trace Logging
Is a refined form of active logging: you can view information about a single
subscriber session
Used to gain more information about user sessions
Implication is that user session is active
Session to be traced can be identified by any of the following:
Call-id
IP address
Mobile Stations Identifier (msid)
Subscriber username
(trace logging active)
CLI user 1
Output sent to system log
System Log (buffer)

Does affect system performance

Anyone with operator privilege (or above) can do it:
logging trace msid 9785551212
To turn off trace logging: no logging trace <msid> (or exit from CLI
session)

Do Not Distribute
13
3-13
Monitor Logging
Monitor logging records all activity associated with a particular session. This
includes call setup.
Unlike trace logging, monitor logging is a command entered while in configuration
mode. This means that that monitor becomes part of the configuration file, and will
survive system re-boots. It also means that the monitor is always running,
regardless of how many CLI sessions are active.
The output of monitor logging goes to the system log (buffer), and any associated
syslog servers.
This functionality is primarily available in order to comply with law enforcement
agency requirements for monitoring capabilities of particular subscribers.
Monitors can be performed based on a subscribers MSID, username or IP address:
Keyword/Variable
Description
msid ms_id
Indicates a monitor log will be generated for a session identified by the mobile
station identification (MSID) number. ms_id must be from 7 to 16 digits
specified as an IMSI, MIN, or RMI.
username
username
Indicates a monitor log will be generated for a session identified by the

username. username is the name of subscriber that was previously
configured.
ip_addr
Specific IP address.
ipv6_addr
Special IPv6 address
3-14

Do Not Distribute
Monitor Logging
Like trace logging, used to obtain detailed information about sessions, but
includes call setup
Output goes to active buffer and syslog server
Can only be started by user with administrator privilege
Not dependent on a CLI instance to run
started in config mode
runs in background
can be save to config file (survive reboots)
Does affect performance
[local]Training# config
[local]Training(config)# logging monitor username
remoteuser2
CLI user 1
(monitor logging active)
Buffer
(to syslog server)

Do Not Distribute
15
3-15
Crash Logging
In the event of a software crash, the system stores information that could be useful
in determining the reason for the crash. This information can be maintained in
system memory or it can be transferred and stored on a network server.
The system supports the generation of the following two types of logs:
Crash log - Crash logs record all possible information pertaining to a

software crash. Due to their size, they cannot be stored in system memory.
Therefore, these logs are only generated if the system is configured with a
Universal Resource Locator (URL) pointing to a local device or a network
server where the log can be stored.
Abridged crash log - These logs are automatically generated when a

software crash occurs and are stored in system memory. The abridged crash
log contains a subset of the possible information that could be generated with
a crash log. These logs are generated even if a full crash log is generated
and can be viewed using the CLI.
The abridged crash log will always be available. Abridged crash logs are stored on
the CompactFlash on the SPC/SMC. They are located in the /flash/crsh2/ directory
with file names in the mc-slot-cpu-pid-xxxxxxxx format. You can see a list with the
following command:
show crash list
Details of one of the entries on the list can be obtained with the following CLI
command:
show crash number <crash_number>
Details of configuring the system to store an un-abridged crash log are shown on
the next slide.
3-16

Do Not Distribute
Crash Logging
Stores information about software crashes on the system
Used for debugging by Cisco engineering
Two flavors:
Abridged (mini-core) crash log automatically configured
and stored in memory on SMC module
Un-abridged (core) crash log core; must be configured and
is stored on an external device (CF, PCMCIA or network
server)
Compact Flash on SMC/SPC
Process
Crash
abridged
(default)
/crsh2/
mc-xx-xx-xxxxx.xxxxxxx
CLI: show crash list

CLI: show crash number <#>
Un-abridged
(requires configuration)
CF, PCMCIA or network

Do Not Distribute
17
3-17
Configuring Crash Logging

The system can be configured to store software crash log information to any of the
following locations:
CompactFlash - Installed on the SPC/SMC
PCMCIA Flash Card - Installed in either the PCMCIA1 or PCMCIA2 slots on

the SPC or in the PCMCIA1 slot on the SMC
Network Server - Any workstation or server on the network that the system
can access using the Trivial File Transfer Protocol (TFTP), the File Transfer
Protocol (FTP), the Secure File Transfer Protocol (SFTP), or the Hyper-Text
Transfer Protocol (HTTP); this is recommended for large network
deployments in which multiple systems require the same configuration.
The table below provides some details of the command shown on the opposite
page:
Keyword/Variable
Description
encrypted
The specified url is in encrypted format.
filename-pattern
pattern
The filename-pattern is a string containing any or all of the following

variables:
%hostname% - The system hostname.
%ip% - A SPIO IP address (v4 only)
%cpu% - CPU number
%card% - Card number
%time% - POSIX timestamp in irritating hex notation %filename% - Alias for
crash-%card%-%cpu%-%time-core%
%% - A single % sign
If no pattern is specified the result is the same as the pattern filename.
url crash_url
Specifies the path and name that the configuration file is to be stored to.
crash_url represents the Universal Resource Locator for the configuration
file and can be expressed in one of the following formats:
{ /flash | /pcmcia1 | /hd }/directory/file_name
file:/{ /flash | /pcmcia1 | /hd } /directory/file_name
tftp://{ ipaddress | host_name }/directory/file_name
ftp://[ username [ :password ] @ ] { ipaddress |
host_name }/directory/file_name
sftp://[ username [ :password ] @ ] host [ /path ] /filename
restrict mbyte
Specifies the maximum amount of memory to use for storing crash files on a
local device. This keyword should only be used if the URL points to a local
device.
The default is 128 megabytes.
3-18

Do Not Distribute
Configuring Crash Logging

To configure the system for storage of an un-abridged
crash log, use the following command:
crash enable [ encrypted ] url crash_url [
filename-pattern pattern ] [ restrict mbyte ]
Compact Flash on SMC/SPC
Process
Crash
abridged
(default)
/crsh2/
mc-xx-xx-xxxxx.xxxxxxx
Un-abridged
(requires configuration)
CF, PCMCIA or network

Do Not Distribute
19
3-19
Viewing Logging Configuration

In order to obtain a list of the systems logging configuration, you can use the show
logging command.
This will display a number of things:
3-20
Runtime logging filter settings a list of all facilities and their present filter
setting.
Syslog configuration if any syslog servers are configured, this will be shown.
Trace logging if any trace log is in place, this will be listed.
Disabled Event logs certain logging events can be disabled so that they will
not be seen in the system log. If configured, the disabled event IDs will be
shown.
Console logging state normally, all entries in the system log are also
displayed at the console; therefore the default setting is enabled.
Configure logging monitors if a monitor log has been configured for any
subscribers, it will be listed here.

Do Not Distribute
Viewing Logging Configuration

To get a complete view of all logging configuration,
use the show logging command
CLI: show logging

Active Buffer
Runtime logging filters
(to syslog server)
Capacity: 50,000 events
Syslog configuration
Trace logging configured
Disabled event logs
Console logging state
Configured logging monitors
CLI sessions

Do Not Distribute
21
3-21
Viewing the System Log

Event logs generated by the system can be viewed in one of the following ways:
From the syslog server - If the system is configured to send logs to a syslog
server, the logs can be viewed directly on the syslog server.
From the system CLI - Logs stored in the system memory buffers can be
viewed directly from the CLI.
From the console port - By default, the system automatically displays

events over the console interface to a terminal provided that there is no CLI
session active.
There are two memory buffers that store event logging information. The first buffer
stores the active log information. The second buffer stores inactive logging
information. The inactive buffer is used as a temporary repository to allow you to
view logs without having data be overwritten. Logs are copied to the inactive buffer
only through manual intervention.
Use the show logs command to view the system event log:
Keyword/Variable
Description
active
Indicates output is to display data from active log memory buffer.
inactive
Indicates output is to display data from inactive log memory buffer.
callid call_id
Specifies a call ID for which log information is to be displayed. call_id

must be specified as a 4-byte hexadecimal number.
facility
facility
Specifies the facility to log information for. A facility is a protocol or task that
is in use by the system. Refer to the Supported Logging Facilities section for
a complete list of facilities that can be used with this keyword.
level
report_level
following: critical, error, warning, unusual, info, trace, debug
since
from_date_time
Specifies that only logs generated after a specific time (from_date_time) are
to be displayed.
from_date_time must be formatted as YYYY:MM:DD:HH:mm or
YYYY:MM:DD:HH:mm:ss.
until
to_date_time
Used in conjunction with the since keyword to specify a particular time

duration over which logs were collected. to_date_time specifies the upper
time limit of the range.
to_date_time must be formatted as YYYY:MM:DD:HH:mm or
YYYY:MM:DD:HH:mm:ss. to_date_time must be a time which is more recent
than the from_date_time variable used with the since keyword.
3-22

Do Not Distribute
Viewing Event Logs (runtime)

You can view the contents of the systems event log from:
A syslog server
CLI session
Console port
From CLI or console, you can view active or inactive buffers
show logs active (default)
show logs inactive
Logged events are only stored in active buffer
CLI: show logs
Active Buffer
(to syslog server)
CLI: show logs inactive

Inactive buffer

Do Not Distribute
23
3-23
Logs Checkpoint Command

Both active and inactive event log memory buffers can be viewed using the CLI.
However, it is preferable to view the inactive log in order to prevent any data from
being over-written. The information from the active log buffer can be copied to the
inactive log buffer by entering the following command:
logs checkpoint
When the active log memory buffer is copied to the inactive log memory buffer
existing information in the inactive log memory buffer is deleted
3-24

Do Not Distribute
Logs Checkpoint Command

Sometimes it is useful to pull logging entries out of the system
buffer before they are overwritten
The logs checkpoint command does this
Contents of active buffer are moved to inactive buffer (clearing
active buffer)
Any data in inactive buffer is overwritten (lost) by this operation
- Runtime Logging
- Active Logging
(to syslog server)

Active Buffer
CLI: show logs active
CLI: logs checkpoint

Inactive Buffer
CLI: show logs inactive


Do Not Distribute
25
3-25
Disabling Event Log Entries

The system provides the ability to restrict the sending of a specific event ID or a
range of event IDs to minimize the amount of data logged to that which is most
useful.
To configure event ID restrictions, enter the following command:
logging disable eventid event_id [ to eventid_last ]
Identification numbers (IDs) are used to reference events as they occur when
logging is enabled on the system. Each facility possesses its own range of event IDs
as indicated in a the sample table below:
Table 12-2 System Event Facilities and ID Ranges
Facility
Event ID Range
Active Charging Service (ACS) Controller Events
90000-90999
Active Charging Service (ACS) Manager Events
91000-91999
Command Line Interface Facility Events
30000-30999
Content Steering Service Facility Events
77000-77499
DHCP Facility Events
53000-53999
Event Log Facility Events
2000-2999
GCDR Facility Events
66000-66999
GTPC Protocol Facility Events
47000-47999
GTPC Signaling Demultipler Manager Facility Events 46000-46999

GTPP Storage Server GCDR Facility Events
98000-98099
GTPU Protocol Facility Events
45000-45999
IP Access Control List (ACL) Log Facility Events
21000-21999
IP Route Facility Events
20000-20999
IPSec Protocol Facility Events
55000-55999
L2TP Manager Facility Events
48000-48999
Lawful Intercept Log Facility Events
69000-69999
Multicast Proxy Facility Events
94000-94999
NPU Manager Facility Events
17000-17999
Point-To-Point Protocol Facility Events
25000-25999
RADIUS Authentication Protocol Facility Events
23000-23999
Session Manager Facility Events
10000-12999
SNMP Protocol Facility Events
22000-22999
System Facility Events
1000-1999
3-26

Do Not Distribute
Disabling Log Event Entries

Sometimes it is useful to not log something
Insignificant events that occur regularly can be ignored by issuing
the following commands:
config
logging disable <event-id>
Only someone with administrator privileges can do this

Event blocked here
System facilities
aaaclient
aaamgr
aaaproxy
.
.
sessmgr
user-data
vpn
(to syslog server)

Buffer

Do Not Distribute
27
3-27
3-28

Do Not Distribute
28
ST16/ASR5000
Alarms and Traps
Lesson 4

Do Not Distribute
4-1
Lesson Overview
In this lesson, you are shown the most efficient commands to use in order to view
alarms and/or traps on the system.
4-2

Do Not Distribute
Lesson Overview
This module shows important CLI commands for
viewing:
SNMP traps
System alarms

Do Not Distribute
4-3
Alarms Versus Traps

An alarm is a condition that is detected on the system.
As a result of this alarm condition being detected, an SNMP trap message is sent to
an external trap receiver
An SNMP trap is a notification sent from the SNMP agent on the ST16/ASR5000 to
an external SNMP manager. It is not solicited by the SNMP Manager. Also, a trap is
not stored on the flash card of the management module on the ST16/ASR5000, but
is buffered in memory. Up to four hundred traps are stored in memory, and then the
oldest ones are overwritten by the newest ones.
An alarm is also stored in memory, but is removed from memory when the condition
that created it is cleared.
All alarms, when they are asserted and when they are cleared, will produce a
respective trap. However, some traps are not related to any alarm. For instance,
when a new CLI session is started a trap is sent out to indicate this. But no alarm
condition is declared.
4-4

Do Not Distribute
Alarms Versus Traps

There is a difference between alarms and traps:
All alarms on the system are sent as SNMP traps
Not all SNMP traps are generated by alarms
Alarms that have been cleared cannot be viewed; only
outstanding alarms can be viewed
An SNMP trap will occur when the alarm first appears, and will
occur again when the alarm is cleared
Alarms
SSH
SNMP
traps
HP OpenView(tm) or
Other NMS
CLI session

Do Not Distribute
4-5
Viewing Alarms
Alarms do not exist unless they are asserted, or outstanding. Therefore, when you
are viewing any alarm, you are viewing outstanding alarms
From a CLI session, you can use the following command to view alarms and some
details about each one:
show alarm outstanding verbose
A sample output from this command is shown in the slide on the opposite page.
You can view some general alarm-related statistics with the following command:
show alarm all
This command also provides some general statistics on the number of alarms that
have occurred, the state of the audible alarm on the SPC/SMC front plate, and the
state of the central office alarm connector on the SPIO card.
4-6

Do Not Distribute
Viewing Alarms
Use the following command to view alarm-related
information on the system:
show alarm outstanding all verbose
[local]Training# show alarm outstanding all verbose

Severity Object
-------- --------Minor
Port 36/1
Minor
Port 36/2
Minor
Port 36/4
Minor
Port 36/6
Minor
Port 36/8
Timestamp
---------------------------Thursday January 22 07:19:3
Thursday January 22 07:19:3
Alarm ID
----------------------------------5765627287628283904 Port link down
5765627287628283905 Port link down
5765627287628283906 Port link down
5765627287628283907 Port link down
5765627287628283908 Port link down

Do Not Distribute
4-7
Viewing Traps
The best way to view traps that have been sent to a trap manager is to use the
following CLI command:
show snmp trap history
A sample output of this command is shown on the opposite page.
Note that you will not see all traps that have been sent since the system last booted.
You will only see the last four hundred traps. These traps are stored in memory, and
the amount of memory reserved for this purpose is limited.
Anther command that is related to SNMP traps:
show snmp trap statistics
The output of this command shows a cumulative count of traps that have been sent
by the ST16/ASR5000.
4-8

Do Not Distribute
Viewing Traps
Use this command to view traps that have been sent to an
external manager:
show snmp trap history
[local]Training# show snmp trap history
There are 83 historical trap records (400 maximum)
Timestamp
Trap Information
------------------------ ---------------------------------------------------Thu Jan 22 07:17:27 2009 Internal trap notification 5 (CardUp) card 8
Thu Jan 22 07:17:27 2009 Internal trap notification 55 (CardActive) card 8
Thu Jan 22 07:17:27 2009 Internal trap notification 55 (CardActive) card 24
Thu Jan 22 07:17:28 2009 Internal trap notification 93 (CardStandby) card 25
Thu Jan 22 07:17:28 2009 Internal trap notification 93 (CardStandby) card 24
Thu Jan 22 07:17:28 2009 Internal trap notification 36 (PortLinkUp) card 24
port 3 ifindex 402849792

Do Not Distribute
4-9
Alarms Sent as Traps

All alarms, when they are asserted and when they are cleared, will produce a
respective trap. An example of an alarm being asserted, and its associated trap
being sent, is shown on the opposite page. When this alarm is cleared, a trap will
again be sent and the alarm will no longer be seen as an outstanding alarm.
4-10

Do Not Distribute
Alarms Sent as Traps

This shows how an outstanding alarm and its respective
SNMP trap
If this port were to come up, another trap would be sent
and this entry would be removed from the outstanding
alarms list
[destination-1]training# show alarm outstanding verbose
Severity Object
Timestamp
Alarm ID
------ ---------- ---------------------------
---------------
Minor
5765605264513892352 Port link down
Port 36/8
Friday January 16 12:57:13
An alarm sent as a trap

[destination-1]training# show snmp trap history | grep -i port
Fri Jan 16 12:57:13 2009 Internal trap notification 35 (PortLinkDown)
card 36 port 8 ifindex 604504064

Do Not Distribute
11
4-11
Viewing Alarms on the Web Element Manager

On the Web Element Manager, alarms are stored in a database. So you can view
not only outstanding alarms, but an alarm history.
The table below provides a brief description of the various alarm-related choices
that are shown on the opposite page:
View
Description
Current
Alarm View
This dialog box displays SNMP events and alarms received from all managed
systems that are configured to report fault information to the EMS.
Historical
Alarm View
Allows you to browse the alarm database on the Web Element Manager for
alarms that have occurred in the past
The Pending Alarm View function performs alarm correlation between incoming
SNMP Events/Traps, denoting a cleared state or an alarm state. If an SNMP trap
such as "Link Down" is received by the EMS server and no corresponding
View Pending clearing SNMP event (in this case Link Up) is received, then the Link Down
trap displays in the Pending Alarm View window. When the Link Up trap is
Alarms
received, the EMS server correlates this clearing event with its corresponding
Link Down event and then automatically removes the event from the Pending
Alarm View. The particular alarms are defined and stored in a configuration file
located on the EMS server. These alarms are not user definable.
Alarm
Information
Shows outstanding alarms and alarm statistics on selected ST16/ASR5000

chassis. Equivalent to the CLI command show alarm all
View
Outstanding
Alarms
Shows only outstanding alarms. Equivalent to the CLI command show alarm
outstanding.
Outstanding
Alarm Statistics
Show alarm statistics for a particular chassis. Equivalent to the CLI command
show alarm statistics.
Alarm History
Lists all alarms received from a particular chassis
CO Alarm
Configuration
Shows the present state of the chassis audible alarm and the Central Office
alarm outputs.
Audio Alarm
Scheduler
Provides a schedule for disabling audio alarm on the WEM for specific severity
levels of traps
Audio Alarming
Enable/Disables audio alarm on WEM.
4-12

Do Not Distribute
Viewing Alarms on the Web Element

Manger
Available alarm/trap displays

Do Not Distribute
13
4-13
4-14

Do Not Distribute
14
Lab: Troubleshooting the System
Lab Overview
This lab consists of a series of exercises that allow you to execute various CLI
commands that are related to troubleshooting the system.
Upon completing this lesson, you will be able to:
Identify defective hardware by viewing LEDs.

Test system and central office alarms.
Use various commands to test network connectivity to/from the system and
its ancillary devices.
Perform system diagnostics to determine the health of the system.
Obtain hardware status information.
Exercise 1: Detecting Faulty Hardware Using Component LEDs

Upon applying power to the chassis, power is provided to the upper and lower
fan trays, and every application and line card that is installed.
Each PFU, application, and line card installed in the system has light emitting diodes
(LEDs) that indicate its status. This section provides information and instructions
pertaining to using LEDs to verify for verifying that all of the installed components
are functioning properly.
Using the CLI to View Component LEDs

The status of application and line card LEDs can be viewed through the systems
CLI by entering the following command:
show leds all
The status of the chassis two Power Filter Units (PFUs) can be viewed by entering
the following command:
show power chassis

Do Not Distribute
4-15
Exercise 2: Testing System Alarm Outputs

The system provides the following two physical alarm mechanisms:
System Audible Alarm: Located on the SPC/SMC, the speaker is used to
provide an audible indicator that an alarm has occurred.
CO Alarms Interface: Located on the SPIO, this interface provides a 10-pin
connector that enables three normally-closed dry-contact relays for the
triggering of external audio and/or visual alarm indicators. These indicators
can be used to alert that either a minor, major, or critical alarm has occurred.
The operation of these alarms can be tested by issuing the following command:
test alarm { audible | central-office [ critical | major |
minor ] }
Keyword/Variable
Description
audible
Tests the CO Alarm Speaker on the SPC/SMC to
verify operation.
Tests the CO Alarm Interface on the SPIO to verify
operation. Individual alarms can be tested by using
one of the following keywords:
- critical: Specifies that the critical CO Alarms
output is to be tested.
central-office
- major: Specifies that the major CO Alarms output
is to be tested.
- minor: Specifies that the minor CO Alarms output
is to be tested.
If no keyword is specified, all alarms will be tested.
When this command is executed, the specified alarm will be activated for a period of
10 seconds. After this time, the alarm will return to its previous condition.
4-16

Do Not Distribute
Exercise 3: Verifying Network Connectivity

There are multiple commands supported by the system to verify and/or
troubleshoot network connectivity. Note that network connectivity can only be
tested once system interfaces and ports have been configured and bound.
The commands specified in this section should be issued on a per context basis.
This is due to the fact that contexts act like virtual private networks (VPNs) in that
they operate independently of the others. Therefore, ports, interfaces, and routes
configured in one context cannot be tested from another without additional
configuration.
To switch between contexts you must enter the following command at the Exec
mode prompt
context <context_name>
context_name is the name of the context that you wish to switch to.
Using the Ping Command

Using the ping command verifies the systems ability to communicate with a remote
node in the network by passing data packets between and measuring the response.
This command is useful in verifying network routing and if a remote node is able to
respond at the IP layer. Use the ping command to do the following:
Verify connectivity to the default gateway (192.168.1.1) of the source context
Verify connectivity to the RADIUS server (135.2.214.226)

Do Not Distribute
4-17
The command has the following syntax:

ping <host_ip_address> [ count <num_packets> ] [ pattern
<packet_pattern> ] [ size <octet_count> ] [ src {<
src_host_name> | <src_host_ip_address> } ]
Keyword/Variable
host_ip_address
count
pattern
size
src
Description
The IP address of the remote node which is the target of the ping
command.
Specifies the number of packets to send to the remote host for
verification. num_packets must be within the range 1 through
10000. The default is 5.
Specifies a pattern to use to fill the internet control message
protocol packets with. packet_pattern must be specified in
hexadecimal format with a value in the range hexadecimal 0x0000
through 0xFFFF. packet_pattern must begin with a 0x followed by
up to 4 hexadecimal digits. The default is that each octet of the
packet is encoded with the octet number of the packet.
Specifies the number of bytes each IP datagram. octet_count must
be a value in the range 40 through 18432. The default is 56.
Specifies an IP address to use in the packets as the source node.
src_host_name: specifies the source node using the nodes logical
host name which must be resolved via DNS lookup.
src_host_ip_address: specifies the source node using the nodes
assigned IP address The default is the IP address of the interface
through which the ping was issued.
If no response is received from the target, follow these troubleshooting procedures:
Verify that the correct IP address was entered.

Attempt to ping a different device on the same network. If the ping was
successful then it is likely that your system configuration is correct. Verify that
the device you are attempting to ping is powered and functioning properly.
Verify the port is operational.
Verify that your configuration of the ports and interfaces within the context are
correct.
If your configuration is correct and you have access to the device that you are
attempting to ping, ping the system from that device.
If there is still no response, it is likely that the packets are getting discarded
by a network device. Use the traceroute and show ip static-route commands
discussed in this chapter to further troubleshoot the issue.
Using the Traceroute Command

The traceroute command collects information on the route data will take to a
specified host. This is a useful troubleshooting command that can identify the
4-18

Do Not Distribute
source of significant packet delays or packet loss on the network. This command
can also be used to identify bottlenecks in the routing of data over the network.
The command has the following syntax:
traceroute { <host_name> | <host_ip_address> } [ count
<packets> ] [ df ] [ maxttl <max_ttl> ] [ minttl <min_ttl> ]
[ port <port_number> ][ size <octet_count> ] [ src
{ <src_host_name> | <src_host_ip_address> } ][ timeout
<seconds> ]
Keyword/Variable
host_name
host_ip_address
count
df
maxttl
minttl
port
size
Description
Identifies the remote node to trace the route to by the hostname.
host_name specifies the remote node using the nodes logical
host name which must be resolved via DNS lookup.
Identifies the remote node to trace the route to by the IP address.
host_ip_address specifies the remote node using the nodes
assigned IP address specified using the standard IPv4 dotteddecimal notation.
Specifies the number of UDP probe packets to send. The default
is 3.
Indicates the packets for the tracing of the route should not be
fragmented. If a packet would require fragmenting then it is
dropped and the ICMP response Unreachable, Needs
Fragmentation is received.
Specifies the maximum time to live (TTL), in seconds, for the
route tracing packets. max_ttl must be specified as a value in the
range of 1 through 255. It is an error if max_ttl is less than min_ttl
whether min_ttl is specified or defaulted. The time to live is the
number of hops through the network, i.e., it is not a measure of
time. The default maximum TTL is 30 seconds.
Specifies the minimum time to live, in seconds, for the route
tracing packets. min_ttl must be specified as a value in the range
of 1 through 255. It is an error if min_ttl is greater than max_ttl
whether max_ttl is specified or defaulted. The time to live is the
number of hops through the network, i.e., it is not a measure of
time. The default minimum TTL is 1 second.
Specifies a specific port to connect to where port_number must
be a value in the range 1 through 65535. The default port is
33434.
Specifies the number of bytes each packet. octet_count must be
a value in the range 40 through 32768. The default is 40.

Do Not Distribute
4-19
Keyword/Variable
src
timeout
Description
Specifies an IP address to use in the packets as the source
node. src_host_name: specifies the remote node using the
nodes logical host name which must be resolved via DNS
lookup. src_host_ip_address: specifies the remote node using
the nodes assigned IP address specified using the standard
IPv4 dotted decimal notation. The default is the IP address of the
interface through which the ping was issued.
Specifies the maximum time to wait for a response from each
route tracing packet. seconds must be a value in the range 2
through 100. The default is 5.
Viewing IP Routes
The system provides a mechanism for viewing route information to a specific node
or for an entire context. This information can be used to verify network connectivity
and to ensure the efficiency of the network connection. The command has the
following syntax:
show ip route [<route_ip_address> [<route_gw_address>]]
Keyword/Variable
route_ip_address
route_gw_address
Description
Specifies the IP address of a network node for which route
information will be displayed. route_ip_address must be
expressed in dotted-decimal notation.
Specifies the IP address of the gateway router between the
system and the
network node for which route information will be displayed.
This is an optional
keyword. route_gw_address must be expressed in dotteddecimal notation.
If no keywords are specified, all IP routes within the contexts routing table will be
displayed.
4-20

Do Not Distribute
Viewing the Address Resolution Protocol Table

The system provides a mechanism for viewing Address Resolution Protocol (ARP)
table information to a specific node or for an entire context. This information can be
used to verify that when the system sends an ARP packet, it receives valid
responses from other network nodes. The command has the following syntax:
show ip arp [ arp_ip_address ]
arp_ip_address specifies a specific network node for which to display ARP
information. If this keyword is not specified, all entries within the contexts ARP table
will be displayed.

Do Not Distribute
4-21
Exercise 4: Using the Systems Diagnostic Utilities

The system provides protocol monitoring and test utilities that can be useful when
troubleshooting or verifying configurations. The information generated by these
utilities can, in many cases, either identify the root cause of a software or network
configuration issue or, at the very least, reduce the number of possibilities.
This exercise contains information and instructions for using these utilities. Using
the Protocol Monitor
For troubleshooting purposes, the system provides a protocol monitoring utility. This
tool can be used to display protocol information for a particular subscriber session or
for every session being processed.
The monitor tool is intrusive in that it may cause session processing delays and/or
data loss. Therefore it should be used only when troubleshooting.
The systems protocol monitor displays information for every session that is
currently being processed. Depending on the number of protocols monitored, and
the number of sessions in progress, a significant amount of data will be generated. It
is highly recommended that logging be enabled on your terminal client in order to
capture all of the information that is generated.
Follow the theses steps to invoke and configure the protocol monitoring tool:
1. Invoke the protocol monitor by entering the following command:
monitor protocol
2. Choose the protocol that you wish to monitor by entering the
number associated with the protocol at the Select: prompt. A
greater-than sign ( > ) appears next to the protocol you selected.
3. Repeat step 2 as needed to choose multiple protocols.
4. Enter B to begin the protocol monitor. If you selected any protocol
other than 11 (SNMP), the following message is displayed:
WARNING!!! You have selected options that can DISRUPT USER SERVICE
Existing CALLS MAY BE DROPPED and/or new CALLS MAY
FAIL!!!
(Under heavy call load, some debugging output
may not be displayed)
Proceed? - Select (Y)es or (N)o
4-22

Do Not Distribute
5. Enter Y to proceed with the monitor or N to go back to the previous menu.

6. Configure the amount of information that is displayed by the monitor. To enable
or disable options, enter the letter associated with that option (C, D, E, etc.). To
increase or decrease the verbosity, use the plus ( + ) or minus ( - ) keys.
The current state, ON (enabled) or OFF (disabled), is shown to the right of each
option.
7. Press the Enter key to refresh the screen and begin monitoring.

Do Not Distribute
4-23
Using the Protocol Monitor for a Specific Subscriber

The systems protocol monitor can be used to display information for a specific
subscriber session that is currently being processed.
1. To invoke the session-specific protocol monitor from the command line and
enter directly into the monitoring mode, enter the following command:
monitor subscriber [callid call_id | imsi imsi | ipaddr
ip_address
|
msid
ms_id
|next-call
|
username
subscriber_name]
Keyword/Variable
callid
imsi
ipaddr
msid
next-call
username
4-24
Description
Specifies that the monitor is executed for a subscriber
with a specific call identification number (callid). call_id
is the specific call identification number that you wish to
monitor. This must be an 8-byte Hexadecimal number.
with a specific International Mobile Subscriber Identity
(IMSI) number. imsi is the specific International Mobile
Subscriber Identity number that you wish to monitor.
This must be an IMSI number with a maximum of 15
digits.
with a specific IP address. ip_address is the specific IP
address that you wish to monitor entered in IPv4 dotted
decimal notation (###.###.###.###).
with a specific mobile station identification (MSID)
number. ms_id is the specific mobile station
identification number that you wish to monitor. This
must be an MSID number with a maximum of 15 digits.
Monitor the next call made to the system across all
active services.
with a specific username. subscriber_name is the
specific username that you wish to monitor. The
username must be a string of 1 to 127 alpha and/or
numeric characters.

Do Not Distribute
2. If no session matching the specified criteria was being processed when the
monitor was invoked, NO MATCHING CALL - waiting for a matching call to
connect... is displayed.
3. Configure the amount of information that is to be displayed by the monitor. To
enable or disable options, enter the letter associated with that option (C, D, E,
etc.). To increase or decrease the verbosity, use the plus ( + ) or minus ( - )
keys.
option.
4. Repeat step 3 as needed to enable or disable options.
5. Choose the protocols that you wish to monitor by entering the number
associated with the protocol (11 through 19 or 21 through 23).
protocol.
6. Repeat step 5 as needed to enable or disable multiple protocols.
7. Press the Enter key to update the menu screen

Do Not Distribute
4-25
Testing a RADIUS Authentication Server

The CLI provides a mechanism for testing network connectivity with and
configuration of RADIUS authentication and accounting servers. This function is
useful in determining the accuracy of the systems RADIUS configuration, the
configuration of the subscriber profile on the RADIUS server, and troubleshooting
the servers response time.
When used to test a RADIUS authentication server, the tool generates an
authentication request message for a specific username.
IMPORTANT: The username must already be configured on the RADIUS
authentication server prior to executing the test.
In the training lab, use the following username/password: remoteusr2/password.
Note: for this exercise, the radius servers IP address will be 135.2.214.226. Please
make sure you are executing the test while you are in the same context where the
AAA logical interface is configured.
To execute the RADIUS authentication test tool, enter the following command:
radius test authentication { all | server <server_name> port
<server_port> } <user_name> <password>
Keyword/Variable
Description
all
Specifies that all configured RADIUS authentication
servers should be tested.
Specifies the IP address of a specific RADIUS
server
authentication server to test.
Specifies the UDP port over that the system should
port
use when communicating with the RADIUS
authentication server to test.
Specifies a username that will be supplied to the
username
RADIUS server for authentication.
Specifies the password associated with the
password
username that will be supplied to the RADIUS
server for authentication.
4-26

Do Not Distribute
The following is a sample of this commands output for a successful response when
testing a RADIUS authentication server with an IP address of 192.168.250.150 on
port 1812.
Authentication from authentication server 192.168.250.150,
port 1812
Authentication Success: Access-Accept received
Round-trip time for response was 8.8 ms
Exercise 5: Monitoring System Hardware Status

This exercise provides information on monitoring the status of system hardware
using the show command accessible in the systems command line interface
(CLI). The following topics are covered:
Viewing Power System Status
Viewing Fan Tray Status
Viewing Installed Card Status

Do Not Distribute
4-27
Viewing Power System Status

The chassis receives power from two load-sharing and redundant Power Filter
Units (PFUs). The status of the PFUs can be viewed by entering either of the
following command:
show power chassis
This command displays the status of each of the chassis two PFUs.
As the system progresses through its boot process, power is applied to the chassis
slots in which cards are installed. Empty slots will not receive power. The power
status of the individual chassis slots can be viewed by entering the following
command:
show power { <slot_number> | all }
Keyword/Variable
slot_number
all
Description
Specifies an individual slot in the chassis for
which to display power status.
Specifies that the power status of all chassis
slots should be displayed.
Viewing Fan Tray Status

The chassis is equipped with both upper and lower fan trays. The lower fan tray
draws air into and pushes it up through the chassis. The upper fan tray is used to
exhaust the air through vents at the upper-rear of the chassis.
The following command displays the fan tray state, speed (high or low), and
temperature for the upper and lower fan trays:
show fans
Viewing Installed Card Status

The CLI provides multiple commands for monitoring the status of application
and line cards. This includes such information as:
Listing Installed Cards
Line Card-to-Application Card Mappings
Diagnostics
Runtime Information
LED status
Port Information.
4-28

Do Not Distribute
Viewing Installed Application and Line Cards

The chassis has 16 front-loading slots used to host application cards and 32 rearloaded slots used to host line cards.
The following command can be used to list the application cards that are currently
installed:
show card table
Additionally, this command displays any rear-installed Redundancy Crossbar
Cards (RCCs), used to provide PAC/PSC-to-Line Card redundancy.
This command also displays the operational state of each installed application
card, whether or not the card is a single point of failure, and the line cards it is
attached to.
The following command can be used to list the rear-installed line cards:
show linecard table
In addition to listing the installed line cards, this command also displays the
operational state, the number of available ports, whether or not the card is a single
point of failure, and the front-installed application card it is attached to.
Application and line cards have the following operational states:
Active: Indicates that the card is an active component that will be used to
process subscriber data sessions.
Standby: Indicates that the card is a redundant component. Redundant
components will become active through manual configuration or
automatically should a failure occur.
Ready: Indicates that an Ethernet 10/100 or Ethernet 1000 line card is
installed behind a PAC/PSC in Standby mode.
Offline: Indicates that the card is installed but is not ready to process
subscriber data sessions. This could be due to the fact that it is not
completely installed (i.e. the card interlock switch is not locked) or that its
processes have been halted.

Do Not Distribute
4-29
Viewing Line Card-to-Application Card Mappings

The system automatically maps, or associates, rear-installed line cards to frontinstalled application cards upon installation. This mapping provides the system with
a type of internal routing table that allows it to determine how to get data back to the
user.
To view the card mapping table, enter the following command:
show card mappings
The mapping table also displays the type of association that exists between the
cards. The following association types are supported:
Direct: The line card is operating in conjunction with the application card
installed directly in front of it.
Cross: The SPIO installed in slot 24 is operating in conjunction with the
SPC/SMC installed in slot 9 or the SPIO in slot 25 is operating in conjunction
with the SPC/SMC in slot 8.
IMPORTANT: Cross mappings will only occur if the SPC/SMC that the SPIO was
formerly operating behind became disabled (either automatically due to an error,
or through manual configuration).
4-30
RCC 40: A line card (non-SPIO) installed in chassis slots 17 through 23 or 26

through 32 is operating in conjunction with a PAC/PSC installed in a chassis
slot that is not directly in front via the RCC in slot 40.
RCC 41: A line card (non-SPIO) installed in chassis slots 33 through 39 or 42
through 48 is operating in conjunction with a PAC/PSC installed in a chassis
slot that is not directly in front via the RCC in slot 41.

Do Not Distribute
Performing a Hardware Inventory

Detailed information about each of the cards and the major components used on
them can be displayed using the CLI as part of hardware inventory. The inventory
provides such information as serial numbers, part numbers, hardware revision, and
firmware versions.
The following table describes the commands supported by the CLI to perform an
inventory of the cards installed in the system.
Command
show hardware
inventory
show hardware version

board
show hardware card

<slot_number>
Description
This command provides a table that lists all of the
cards installed in the
chassis and their hardware revision, part, serial,
assembly, and fabrication numbers.
This command provides a table that lists all of the
cards installed in the
chassis and their hardware revision, and the
firmware version of the
on-board Field Programmable Gate Array
(FPGAs).
This command is used to provide detailed
information about a specific card. It reports all of
the same information for the card as the previous
two commands and also includes the cards
description, and information on the CPUs (if
executed for a PAC/PSC or SPC/SMC).
Viewing Card Diagnostics

Boot, power and temperature diagnostics can be viewed for individual cards by
entering the following command:
show card diag <slot_number>
slot_number is the chassis slot number in which the card you want to view
diagnostics for is installed. Viewing Card Runtime Information
Runtime, or real time, information such as operational state, temperature, voltage
conditions, and LED state can be viewed for individual cards by entering the
following command:
show card info <slot_number>
slot_number is the chassis slot number in which the card you want to view
information on is installed.

Do Not Distribute
4-31
Viewing Available Physical Ports

The number and type of physical ports that are available to the system is a function
of the number and type of line cards installed. Earlier in this lesson, information was
provided on using the show linecard table command to view installed linecards.
The CLI also provides a mechanism for displaying the ports that are available.
To view the ports that are available to the system, enter the following command:
show port table
In addition to listing the available ports, the following information is also provided for
each of the ports:
Administrative State: Indicates whether or not the card has been configured
for use via software. If it has been configured, Enabled will be displayed. If
not, Disabled will be displayed.
Operational State: Indicates the operational state of the card, either Up or
Down.
Link Status: Indicates the link status of the network connection, either Up or
Down.
Card State: Indicates the operational mode of the card that the port belongs
to.
Redundancy: Displays the redundant port for that port if one is available.
Viewing Detailed Port Information

In addition to the information displayed by the show port table command, more
detailed information can be viewed for each of the available ports. The type of
detailed port information is:
Port configuration: Indicates the configured duplex mode and speed of the
port.
MAC Address: Indicates the media access control (MAC) address of the
port.
Link information: If the port was configured to automatically detect the
duplex mode and speed, the actual duplex mode and speed will be
displayed.
4-32

Do Not Distribute
Detailed port information can be viewed by entering the following command:

show port info <slot_number>/<port_number>
Keyword/Variable
slot_number
port_number
Description
Specifies the number of the chassis slot in
which the card containing the
port is located and can be any integer value
between 17 through 39 and
42 through 48.
Specifies the number of the port that you wish
view information for. For
example, 24/1 would represent the first port on
the SPIO installed in slot
24.
Viewing System CPU Resource Information

Central processing unit (CPU) resource information can be viewed by entering the
following command:
show resource cpu
Executing this command displays statistics for all active and standby CPUs
available in the system on either installed PACs/PSCs or SPCs/SMCs.
Show Support Details

The show support details command automatically collects the output information
from each of the systems individual show commands and compiles it into a single
listing that is useful for troubleshooting.
The show support details output can be displayed on your console or written to a
tar file that can be stored locally (flash) or remotely on an FTP server. To display the
output on your console, enter the following command at the Exec mode prompt:
show support details

Do Not Distribute
4-33
To display the output on your console and write it to a tar file, enter the following
command at the Exec mode prompt:
show support details to file <url>
Keyword/
Variable
Description
Specifies the location where a tar file with the support detail
information should be created. url may refer to a local or a remote
file. url must be entered using one of the following formats:
to file <url>
ST16:
[ file: ] { /flash | /pcmcia1 | /pcmcia2 } [ /directory ]/file_name
tftp://{ host[ :port# ] } [ /directory ]/file_name
[ ftp: | sftp: ]//[ username[ :password ]@ ]
{ host }[ :port# ][ /directory ]/file_name
ASR5000:
[ file: ] { /flash | /pcmcia1 | /hd }[ /directory ]/file_name
tftp://{ host[ :port# ] }[ /directory ]/file_name
[ ftp: | sftp: ]//[ username[ :password ]@ ]
{ host } [ :port# ] [ /directory ]/file_name
directory is the directory name.
filename is the actual file of interest.
username is the user to be authenticated.
password is the password to use for authentication.
host is the IP address or host name of the server.
port# is the logical port number that the communication protocol is
to use.
If the filename is not specified with a .tar extension, it is
automatically appended to the filename when the file is created
and a message is generated.
The content of the tar file is a follows:

support_summary - An ASCII text file that contains the support detail
information.
information.minicores.tar - A tar file that contains any minicore files found
on the system. Minicore files contain memory core dumps that are captured
during some events. These core dumps provide specific memory locations
and other information about the event. This information is useful to the
technical support team in identifying specifically where an event occurred and
its probably cause.
4-34

Do Not Distribute
CLI Commands That Impact System Performance

Commands that impact system performance fall into two categories:
Category 1Commands that impact performance if executed while the
system is under heavy load.
Category 2Commands that impact performance if multiple instances of the
same or different commands are executed while the system is under heavy
load and simultaneously facilitating multiple CLI sessions.
Category 1 Commands
The commands that impact performance if executed while the system is under
heavy load are as follows:
Hardware Operations: Impact hardware and/or service availability
Exec Mode:
o shutdown
o reload
o card reboot
o card halt
o card restart
o card pac migrate from
o card psc migrate from
Global Config Mode:
o no require pac daughtercard
Logging: (Dependant on facilities selected and system activity, such as number of
subscribers and/or volume of data)
Exec Mode:
o logging filter active facility <XXX> level ( info |
trace | debug )
o logging trace (username <name> | msid <#> | ipaddr
<addr> )
Global Config Mode:
o logging filter runtime facility <XXX> level ( info |
trace | debug )
o logging monitor (msid <name | username <name>)
Monitor and Test Functions:
gtp test echo <XXX> <1...1000000> (Depends on number of
subscribers and/or packets)
monitor protocol (This command menu can be used to select logging of
all PDUs in the system across all calls. For subscriber protocols such as
GTP-C, RADIUS, GTP-U, etc. this can significantly impact performance. The
recommended method for doing this is to execute the "monitor subscriber"
command. Options which may impact performance issue a warning to the
user prior to execution.)
debug ip arp (This could impact GRAT-ARP generation performance
depending on system configuration and/or load.)

Do Not Distribute
4-35
Category 2 Commands
4-36
show subscribers full xGSN-service <name>

show subscribers all
show subscribers active
show subscribers summary
monitor subscriber imsi <IMSI>
monitor subscriber username <username>
monitor subscriber ipadd <IP Addr>
show card table
show line table
show port table
show card mappings
show card hardware
show up statistics
show gtp statistics
show map statistics
show configuration
show session subsystem full
show session duration
show session progress
show task resources
show task info
show card diag
timestamps elapsed-time
show sub data high
show version verbose
show admin session id
show session progress
show resource verbose
show cpu table
show cpu info card 8 verbose
show session sub | grep size
show ip pool
show session disc
show radius counter all

Do Not Distribute
ST16/ASR5000
Troubleshooting
Exercise
Lesson 5

Do Not Distribute
5-1
Troubleshooting Lab
In this lab, the instructor is going to insert some configuration-related errors.
The class is going to divide into teams, with each team creating their own GGSN
configuration.
The configuration that is going to be built by each team is shown on the opposite
page.
For the details of putting this configuration in place, consult your lab guide.
5-2

Do Not Distribute
Troubleshooting Lab

Do Not Distribute
5-3
5-4

Do Not Distribute
Lab: Debugging a GGSN on ASR5000

Each Team is going to build the configuration illustrated below. It consists of two
contexts:
A Gn (or Source) context that contains a Gn logical interface interface and a
GGSN service,
A Gi (or Destination) context that contains an APN, AAA configuration, and an IP
Pool.
The picture below shows all the components:

Do Not Distribute
5-5
Creating Contexts
The table below lists the name of the contexts, depending on your team number.
Use this table to create the correct contexts.
Team
1
2
3
4
5
6
7
8
Ingress Context
source-1
source-2
source-3
source-4
source-5
source-6
source-7
source-8
Egress Context
destination-1
destination-2
destination-3
destination-4
destination-5
destination-6
destination-7
destination-8
config
context <ingress context name>
exit
context <egress context name>
end
Creating and Configuring the Gn Interface and Port

configure
context <ingress context of your team>
Using the table below, create the Gn logical interface for your team by entering the
following command:
interface <interface name>
Team
1
2
3
4
5
6
7
8
5-6
Gn Interface Name
20/1-gn
20/2-gn
20/3-gn
20/4-gn
20/5-gn
20/6-gn
20/7-gn
20/8-gn

Do Not Distribute
ip address <address> <mask>

Team
1
2
3
4
5
6
7
8
IP address/mask
192.168.10.11/24
192.168.10.12/24
192.168.10.13/24
192.168.10.14/24
192.168.10.15/24
192.168.10.16/24
192.168.10.17/24
192.168.10.18/24
exit
exit
port ethernet <slot#/port#>
description <port description>
no shutdown
bind interface <logical int name> <logical int
context>
Team
1
2
3
4
5
6
7
8
Slot#/port
#
20/1
20/2
20/3
20/4
20/5
20/6
20/7
20/8
Port
Description
team1-gn
team2-gn
team3-gn
team4-gn
team5-gn
team6-gn
team7-gn
team8-gn
Logical Int
Name
20/1-gn
20/2-gn
20/3-gn
20/4-gn
20/5-gn
20/6-gn
20/7-gn
20/8-gn
Logical Int
Context
source-1
source-2
source-3
source-4
source-5
source-6
source-7
source-8
end

Do Not Distribute
5-7
Creating and Configuring the Gi /AAA Interface and Port
configure
context <egress context of your team>
Team
1
2
3
4
5
6
7
8
Gi Interface Name
29/1-gi-aaa
29/2-gi-aaa
29/3-gi-aaa
29/4-gi-aaa
29/5-gi-aaa
29/6-gi-aaa
29/7-gi-aaa
29/8-gi-aaa
ip address <address> <mask>

Team
1
2
3
4
5
6
7
8
IP address/mask
192.168.2.1/24
192.168.2.2/24
192.168.2.3/24
192.168.2.4/24
192.168.2.5/24
192.168.2.6/24
192.168.2.7/24
192.168.2.8/24
exit
exit
5-8

Do Not Distribute
port ethernet <slot#/port#>

no shutdown
bind interface <logical int name> <logical int
context>
Team
1
2
3
4
5
6
7
8
Slot#/port
#
29/1
29/2
29/3
29/4
29/5
29/6
29/7
29/8
Port
Description
team1-aaa
team2-aaa
team3-aaa
team4-aaa
team5-aaa
team6-aaa
team7-aaa
team8-aaa
Logical Int
Name
29/1-gi-aaa
29/2-gi-aaa
29/3-gi-aaa
29/4-gi-aaa
29/5-gi-aaa
29/6-gi-aaa
29/7-gi-aaa
29/8-gi-aaa
Logical Int
Context
destination-1
destination-2
destination-3
destination-4
destination-5
destination-6
destination-7
destination-8
end
Creating and Configuring the AAA group and Interface
configure
context <egress context of your team>
Team
1
2
3
4
5
6
7
8
AAA Interface Name

29/1-gi-aaa
29/2-gi-aaa
29/3-gi-aaa
29/4-gi-aaa
29/5-gi-aaa
29/6-gi-aaa
29/7-gi-aaa
29/8-gi-aaa

Do Not Distribute
5-9
ip address <address> <mask> secondary
Team
1
2
3
4
5
6
7
8
IP address/mask
135.2.214.1/24
135.2.214.2/24
135.2.214.3/24
135.2.214.4/24
135.2.214.5/24
135.2.214.6/24
135.2.214.7/24
135.2.214.8/24
end
Creating the GGSN Service in the Source Context

configure
context <ingress context of your team>
ggsn-service <ggsn_service_name>
Team
1
2
3
4
5
6
7
8
5-10
GGSN Service Name

ggsn-svc-1
ggsn-svc-2
ggsn-svc-3
ggsn-svc-4
ggsn-svc-5
ggsn-svc-6
ggsn-svc-7
ggsn-svc-8

Do Not Distribute
In the ggsn-service configuration mode, provision the appropriate parameters from

the table
plmn id mcc xxx mnc xxx
plmn unlisted-sgsn home
Team
1
2
3
4
5
6
7
8
ggsnservice
ggsn-svc-1
ggsn-svc-2
ggsn-svc-3
ggsn-svc-4
ggsn-svc-5
ggsn-svc-6
ggsn-svc-7
ggsn-svc-8
PLMN ID
mcc 201 mnc 301
mcc 202 mnc 302
mcc 203 mnc 303
mcc 204 mnc 304
mcc 205 mnc 305
mcc 206 mnc 306
mcc 207 mnc 307
mcc 208 mnc 308
bind address <address of Gn interface>

Team
1
2
3
4
5
6
7
8
Gn IP Address
192.168.10.11
192.168.10.12
192.168.10.13
192.168.10.14
192.168.10.15
192.168.10.16
192.168.10.17
192.168.10.18
end

Do Not Distribute
5-11
Creating APN in Destination Context

configure
context <destination context)
apn
<apn name>
aaa group <aaa group>

ip address alloc-method local
ip address pool name <pool name of your team>
selection_mode sent_by_ms
authentication pap 1
end
Team Context
containing
APN
1
destination-1
2
destination -2
3
destination -3
4
destination -4
5
destination -5
6
destination -6
7
destination -7
8
destination -8
5-12
APN name
AAA Group
IP Address
Pool name
APN-1
APN-2
APN-3
APN-4
APN-5
APN-6
APN-7
APN-8
aaa-1
aaa-2
aaa-3
aaa-4
aaa-5
aaa-6
aaa-7
aaa-8
pool-1
pool-2
pool-3
pool-4
pool-5
pool-6
pool-7
pool-8

Do Not Distribute
Configuring AAA Parameters (in Destination context)

configure
context <destination context of your team>
aaa group aaa-<your team number>
Team
1
2
3
4
5
6
7
8
AAA group name

aaa-1
aaa-2
aaa-3
aaa-4
aaa-5
aaa-6
aaa-7
aaa-8
radius server 135.2.214.228 key

radius attribute nas-identifier
of your team>
radius attribute nas-ip-address
address of your team>
Team
1
2
3
4
5
6
7
8
secret port 1812

secret port 1812
secret port 1812
<destination context
address <aaa local
AAA Local address

135.2.214.1
135.2.214.2
135.2.214.3
135.2.214.4
135.2.214.5
135.2.214.6
135.2.214.7
135.2.214.8
end

Do Not Distribute
5-13
Configuring IP Address Pools

configure
context <destination context of your team>
ip pool <ip pool name> <pool address range>
Team
1
2
3
4
5
6
7
8
IP Pool
Name
pool-1
pool-2
pool-3
pool-4
pool-5
pool-6
pool-7
pool-8
Pool Address
Range
10.11.0.0/17
10.12.0.0/17
10.13.0.0/17
10.14.0.0/17
10.15.0.0/17
10.16.0.0/17
10.17.0.0/17
10.18.0.0/17
end
Saving and Testing the Configuration

save configuration /flash/<cfg file name> -redundant
Team Saved
Config
Name
1
team1.cfg
2
team2.cfg
3
team3.cfg
4
team4.cfg
5
team5.cfg
6
team6.cfg
7
team7.cfg
8
team8.cfg
Ask your instructor to generate a test call.
To trace your call, prepare to capture the next call using the monitor tool:
monitor subscriber next-call
5-14

Do Not Distribute

Do Not Distribute
5-15
5-16

Do Not Distribute

CS XGSN TS V81 08 10

Transféré par

Informations du document

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

CS XGSN TS V81 08 10

Transféré par

Droits d'auteur :

Formats disponibles

GPRS/UMTS Troubleshooting

Version Number 8.1

Cisco Confidential and Proprietary

Cisco Confidential and Proprietary

Lesson 2: ST16/ASR5000 Software Troubleshooting

Lesson 3: ST16/ASR5000 Logging

Lesson 4: ST16/ASR5000 Alarms and Traps

Lab: Troubleshooting the System

Lesson 5: ST16/ASR5000 Troubleshooting Exercise

Cisco Confidential and Proprietary

Cisco Confidential and Proprietary

Lesson 1: ST16/ASR5000 Hardware Troubleshooting

Lesson 1: ST16/ASR5000 Hardware Troubleshooting

Cisco Confidential and Proprietary

Lesson 1: ST16/ASR5000 Hardware Troubleshooting

Cisco Confidential and Proprietary

Lesson 1: ST16/ASR5000 Hardware Troubleshooting

Cisco Confidential and Proprietary

Lesson 1: ST16/ASR5000 Hardware Troubleshooting

Chassis General View

Cisco Confidential and Proprietary

Lesson 1: ST16/ASR5000 Hardware Troubleshooting

Chassis General View

Upper fan tray

show led all

Fan tray bezels

Lower Fan Tray &

Power filter units

Cisco Confidential and Proprietary

Lesson 1: ST16/ASR5000 Hardware Troubleshooting

Viewing Card-level Details

This field indicates if the software on any of the programmable

The firmware version of the component that allows non-SPC/SMC cards

The version number of one of the Complex Programmable Logic

The firmware version of the Status, Reset, and Monitoring component.

The firmware version of the Chassis Information (CIF) Field

The firmware version of the Power, Status, and Reset component.

The version of the Basic Input/Output System (BIOS).

The firmware version of the Data Transport (DT) Field Programmable

SFP Info (Port

Indicates information pertaining to the small form-factor pluggable (SFP)

Cisco Confidential and Proprietary

Lesson 1: ST16/ASR5000 Hardware Troubleshooting

Viewing Card-level Details

Cisco Confidential and Proprietary

Lesson 1: ST16/ASR5000 Hardware Troubleshooting

Viewing Port-level Status Detail

The description given to the port during software configuration. If no description

The link status, either Up or Down.

Cisco Confidential and Proprietary

Lesson 1: ST16/ASR5000 Hardware Troubleshooting

Viewing Port-level Status Detail

Cisco Confidential and Proprietary

Lesson 1: ST16/ASR5000 Hardware Troubleshooting

Viewing Port-level Counters

The number of overflows received.

The number of frames, less than 64 bytes in length,

The number of frames, less than 64 bytes in length,

The number of frames, with lengths between 64 bytes

The number of frames, larger than the maximum frame

The number of frames, larger than the maximum frame

The number of correct received flow-control frames.

The number of false carrier events detected.