Expert Reference Series of White Papers

Cisco Security
Troubleshooting:
Part III – Intrusion
Prevention Systems

1-800-COURSES www.globalknowledge.com
Cisco Security Troubleshooting: Part III –
Intrusion Prevention Systems
Douglas B. McKillip, P.E., CCSI, CCSP, CCIE #1851

Introduction
This paper is the third in a three-part series of white papers on Cisco security troubleshooting, each of which
examines the challenge of implementing network security on equipment from Cisco Systems while maintaining
the connectivity requirements of the business or enterprise. The focus of this third paper is primarily on trouble-
shooting the proper sensing operation of a signature-based Intrusion Prevention System (IPS). The first two
papers in this series are: Troubleshooting Part I – Connectivity Through ASA or PIX Firewalls and Troubleshooting
Part II – Virtual Private Networks

Troubleshooting Scenario
The following scenario will be used to illustrate many of the concepts discussed in this Cisco security white pa-
per. If the names and IP Addresses look familiar to some of you, this is because the diagram represents a portion
of the lab topology used in Cisco Security classes offered by Global Knowledge.

Perimeter
Router

200.200.1.0/24 Outside-PC
DMZ Subnet 150.150.1.20
172.16.1.0/24

ASA/PIX

DMZ-Srv
172.16.1.15
10.10.10.0.2
NAT: 200.200.1.15

Admin-PC
10.10.10.10

Copyright ©2009 Global Knowledge Training LLC. All rights reserved. 2

Intrusion Prevention Systems
Before we illustrate the effective use of sensor (CLI) commands as well as IPS Device Manager, some basic
principles of operation first need to be discussed. For all deployments of sensor appliances and modules, the
interfaces belonging to these devices can take one of three roles.
1) Command and Control/Management - one per device; only interface w IP Address
2) Promiscuous - multiple per device; sensing interface
3) Inline (as part of a pair) - multiple per device; sensing interface

While the Command and Control/Management interface is used for Management and Monitoring, the Promiscu-
ous or Inline Pair interfaces are the ones into which the packets to be “sensed” arrive. The following sequence of
events occurs with a properly operating and configured signature-based IPS.

1) A packet arrives at a sensing interface (promiscuous or part of an inline pair).

2) The packet is captured by what is known as the sensor app or analysis engine.
3) The sensor app invokes independent signature engines for matching of patterns.
4) If a pattern match is found against the signature database, an alert is generated.
5) Additional deny, blocking, capture, or alert actions are taken, if configured.

The steps outlined above are an oversimplified example of how the IPS operates, assuming that the signature
being matched is enabled and that the alerting behavior is configured. Secondly, the pattern match described in
the fourth step could be the final packet in a multi-packet or fragment stream versus a single “atomic” capture.
This Cisco white paper will focus on troubleshooting the failure of any of the five steps shown above.

Promiscuous Mode Operation – Single Switch with SPAN

port
The diagram to the right represents a modifica-
tion to the troubleshooting scenario depicted
earlier, illustrating the implementation of a
promiscuous sensing interface on an IPS plugged
into a switch. A Switched Port Analyzer (or SPAN
port, as it is commonly known) has been config-
ured for copying packets that enter the switch
port connected to the DMZ interface of the ASA
or PIX firewall.

IOS-based switches must be configured using the

monitor global configuration command using
the following syntax.

Copyright ©2009 Global Knowledge Training LLC. All rights reserved. 3

 monitor1session 1 source interface <hw-interface-name#> both
monitor1session 1 destination interface <hw-interface-name#>

The keyword both is used above to indicate packets being both received and transmitted by the interface. The
commands show monitor and debug monitor can be used to verify proper configuration and operation,
respectively. A sample display of show monitor is shown below.

IPS-Sw# show monitor

Session 1
---------
Type : Local Session
Source Ports :
Both : Fa0/1
Destination Ports : Fa0/4
Encapsulation : Native
Ingress: Disabled

Promiscuous Mode Operation - Multiple Switches with

RSPAN

Occasionally, an IPS will need to be operated in promiscuous mode connected to a switch other than the one
through which the “interesting traffic” is flowing. In this case, a Remote SPAN or RSPAN VLAN can be used to
copy packets from the source switch to this VLAN, carried by a trunk port between the switches, and then to a
SPAN port on the destination switch. Several key steps are required on the two switches to accomplish this.

On both switches:
Switch(config)# vlan 900 (900 is an example; any unassigned VLAN # here is fine)
Switch(config-vlan)# remote-span

Copyright ©2009 Global Knowledge Training LLC. All rights reserved. 4

On Switch2 (see previous diagram):
Switch2(config)# monitor session 1 source interface <hw-interface-name#>
Switch2(config)# monitor session 1 destination remote 900

On Switch1 (see previous diagram):

Switch1(config)# monitor session 1 source remote 900
Switch1(config)# monitor session 1 destination interface <hw-interface-name#>

Once these configurations are in place and spanning-tree for the RSPAN VLAN has been disabled, the reception
of packets by the IPS promiscuous interface can be verified by a simple show interface command.

Sensor1# show interface

Interface Statistics
Total Packets Received = 417
Total Bytes Received = 34867
Missed Packet Percentage = 0
Current Bypass Mode = Auto_off
MAC statistics from interface GigabitEthernet0/0
Interface function = Sensing interface
Description =
Media Type = TX
Default Vlan = 0
Inline Mode = Unpaired
Pair Status = N/A
Hardware Bypass Capable = No
Hardware Bypass Paired = N/A
Link Status = Up
Link Speed = Auto_100
Link Duplex = Auto_Half
Missed Packet Percentage = 0
Total Packets Received = 417
Total Bytes Received = 34867

While this command will verify that the sensor is receiving packets, it does not verify that the bi-directional flow
between each session endpoint is seen. A more useful command on the sensor to verify this “sessionized flow”
is the packet capture command as seen below (which can also be done for inline operation).

Copyright ©2009 Global Knowledge Training LLC. All rights reserved. 5

 Sensor1# packet capture GigabitEthernet0/0
Warning: This command will cause significant performance degrada-
tion
tcpdump: WARNING: ge0_0: no IPv4 address assigned
tcpdump: listening on ge0_0, link-type EN10MB (Ethernet), capture
size 65535 bytes
115 packets captured
115 packets received by filter
0 packets dropped by kernel

Sensor1# packet display packet-file

reading from file /usr/cids/idsRoot/var/packet-file, link-type EN10-
MB (Ethernet)
.....(other output omitted).....
21:24:18.446052 IP 10.10.10.10 > 10.10.10.200: icmp 1208: echo re-
quest seq 3328
21:24:18.447450 IP 10.10.10.200 > 10.10.10.10: icmp 1208: echo re-
ply seq 3328
21:24:19.396929 IP 10.10.10.10 > 10.10.10.200: icmp 1208: echo re-
quest seq 3584
21:24:19.397126 IP 10.10.10.200 > 10.10.10.10: icmp 1208: echo re-
ply seq 3584
21:24:20.403217 IP 10.10.10.10 > 10.10.10.200: icmp 1208: echo re-
quest seq 3840
21:24:20.404216 IP 10.10.10.200 > 10.10.10.10: icmp 1208: echo re-
ply seq 3840
21:24:21.403618 IP 10.10.10.10 > 10.10.10.200: icmp 1208: echo re-
quest seq 4096
21:24:21.403816 IP 10.10.10.200 > 10.10.10.10: icmp 1208: echo re-
ply seq 4096

As can be seen from the trace above of a 1200 byte ping from 10.10.10.10 to 10.10.10.200, both the echo-
request and echo-reply packets can be seen. The observation of both directions of flow is necessary for the IPS
sensor to properly reassemble fragmented packets in addition to its ability to observe “well-behaved” TCP con-
nections (the completion of 3-way handshake & FIN exchange, for example).

Copyright ©2009 Global Knowledge Training LLC. All rights reserved. 6

InLine Mode Operation – Both Interfaces on the Same
Switch

The diagram shown above is a more detailed depiction of the Troubleshooting Scenario seen at the beginning of
this paper. If the two interfaces on an IPS sensor inline pair are connected to the same switch, the DMZ Server
and the ASA/PIX dmz interface must be on different VLANs; otherwise, the sensor will be bypassed. Since each
VLAN is traditionally thought of as its own broadcast domain, this means that VLAN 3 and VLAN 13 now have
been effectively combined into one.

Verification of Sensor Analysis Engine Proper Operation

Now that we have examined ways to confirm the proper capture of traffic by using a promiscuous interface or
an inline pair, our next focus will be on the analysis engine itself, or “sensor app.” There are several ways to
confirm that the analysis engine is running. The first of these is a simple CLI-based command, show version,
whose output is shown below.

show version
Application Partition:

Cisco Intrusion Prevention System, Version 6.0(2)E1

Host:
Realm Keys key1.0
Signature Definition:
Signature Update S280.0 2007-04-11
Virus Update V1.2 2005-11-24
--- (output omitted) ---
MainApp 2007_MAR_29_14_06 (Release) 2007-03-29T14:44:36-
0600 Running
AnalysisEngine 2007_MAR_29_14_06 (Release) 2007-03-
29T14:44:36-0600 Running
CLI 2007_MAR_29_14_06 (Release) 2007-03-29T14:44:36-0600

Copyright ©2009 Global Knowledge Training LLC. All rights reserved. 7

As can be seen previously by the bolded text, the AnalysisEngine is operational. On the next page, an analo-
gous screenshot of IDM is shown of the Monitoring --> System Information area. The display in the text
window also indicates a “running” status.

Verification of Operational Status of Signature Engines

The third step in the five-step sequence of events outlined in the beginning of this paper was the analysis of
captured packet(s) by the various signature engines. A signature engine is a category or “family” of pattern
matches with common tunable thresholds and parameters. An example of this category would be ATOMIC.
IP where the signatures belonging to that “family” all are matched by examining the packet headers of an IP
packet (which would also include the ICMP, UDP, or TCP headers).

As shown by the output below, the CLI command show statistics analysis engine can be used to verify
activity of the signature engines.

show statistics analysis-engine

Analysis Engine Statistics
Number of seconds since service started = 23069
The rate of TCP connections tracked per second = 0
The rate of packets per second = 5

Copyright ©2009 Global Knowledge Training LLC. All rights reserved. 8

 The rate of bytes per second = 1193
----- (output omitted) ----
Inspection Stats
Inspector active call create delete cre-
atePct callPct
AtomicL3L4_IP 4 47461 4 0 0 39
AtomicL3L4_ICMP 1 33 1 0 0 0
AtomicL3L4_TCP 1 46420 1 0 0 38
AtomicL3L4_UDP 1 1008 1 0 0 0
FloodHostICMP 0 33 4 4 0 0
Layer2ARP 1 252 1 0 0 0
MSRPC_TCP 0 3110 622 622 0 2
MSRPC_UDP 1 1008 375 374 0 0
ServiceDnsUdp 1 1008 1 0 0 0
ServiceGeneric 3 47461 3 0 0 39
ServiceHttp 0 622 622 622 0 0
ServiceNtp 2 2016 750 748 0 1
ServiceRpcUDP 1 1008 1 0 0 0
ServiceRpcTCP 1 10672 622 621 0 8
ServiceSnmp 1 1008 1 0 0 0
ServiceTNS 0 1244 1244 1244 1 1
String 1 10672 622 621 0 8
SweepUDP 1 1008 319 318 0 0
SweepICMP 0 33 4 4 0 0
SweepTCP 3 92840 10 7 0 76
SweepOtherTcp 1 46420 4 3 0 38
TrafficIcmp 0 33 2 2 0 0
TrafficIcmpDDOS 0 33 2 2 0 0
TrojanUdp 1 1008 319 318 0 0

Note that even though the documentation on CCO classifies the ATOMIC.IP as an engine, the transport layer
protocols TCP, UDP, and ICMP are separated out statistically. Also note that the line “Number of seconds since
service started” substantiates that the analysis engine is running.

Troubleshooting the Generation of Alerts

Once the determination has been made that the sensor is properly capturing packets and the analysis engine
is running, there are a number of ways to confirm proper signature alert configuration and behavior. One such
method using IDM is depicted below where a signature with SigID 3200 was chosen for editing. Note here that
not only is the signature disabled (which means alerting will be as well), but also the signature is marked as

Copyright ©2009 Global Knowledge Training LLC. All rights reserved. 9

“retired.” This latter term means that the sensor will not dynamically allocate memory resources for the data
structure necessary for scanning for this signature. When “Edit” is clicked, the bottom image results.

In the above window, the Enabled and

Retired Status are changed using the drop-
down menu.

The show events alert CLI command

can be used on a directly connected
terminal or using IP-based SSH terminal
emulators to yield a live monitor. While
this is a useful tool for troubleshooting
the absence of ANY alerts, it would be
practically infeasible in a live production
network due to the volume and speed of
displayed information. A sample output is
shown next for a large ICMP packet.

Copyright ©2009 Global Knowledge Training LLC. All rights reserved. 10

 show events alert
evIdsAlert: eventId=1234900256699266292 severity=informational
vendor=Cisco
originator:
hostId: Sensor1
appName: sensorApp
appInstanceId: 365
time: 2009/02/18 01:32:43 2009/02/18 01:32:43 UTC
signature: description=Large ICMP Traffic id=2151 version=S1
subsigId: 0
marsCategory: DoS/Host
interfaceGroup: vs0
vlan: 0
participants:
attacker:
addr: locality=OUT 10.10.10.10
target:
addr: locality=OUT 10.10.10.200
os: idSource=learned relevance=relevant type=linux
riskRatingValue: attackRelevanceRating=relevant
targetValueRating=medium 35
threatRatingValue: 35
interface: ge0_0
protocol: icmp

The corresponding area of IDM to ob-

serve these alerts (after they occur, not in
real-time) is under the Monitoring Tab,
Events menu selection as seen below.

Another configurable alert property is the

Alert Summarization mode which will
influence the frequency of observed firings
of the signature. As shown on the next
page in the IDM Signature Edit screen for
SigID 1102, the default Summary Mode is
Fire Once. A little known “undocumented
feature” of this alerting mode is that the
frequency of alerts in this mode cannot ex-
ceed once every two minutes; therefore, to
observe the most frequent alerting behav-
ior, Fire All should be chosen.

Copyright ©2009 Global Knowledge Training LLC. All rights reserved. 11

Troubleshooting Addi-
tional Signature Actions
Besides the basic “Produce Alert” default
setting for all Enabled (and also “unRe-
tired”) signatures, additional sensor actions
can be taken, if desired, by configuring these
on an individual signature-by-signature
basis. A comprehensive list of the possible
signature actions is depicted below, taken
from the Configuration --> Policies -->
Signature Definitions section, Actions
button.

Of all the actions depicted in the Assign

Actions dialog box below right, the Request
Block Connection and Request Block Host
options each require that the IPS sensor
initiate either a SSH or a telnet session to
the blocking device (Cisco ASA/PIX, router, or
Catalyst 6500 switch). Once the login profile,
the blocking device properties, and the
blocking interface designation and Access
Control List (ACL) assignments have properly
been configured and applied to the sensor,
it will immediately attempt to establish a ses-
sion with the blocking device. The sensor will
stay logged in for as long as the router/fire-
wall/Catalyst switch is defined as a blocking
device in the IPS configuration. This facilitates
the quick application or removal of the block-
ing access-list entry or shun command, the
latter used by the PIX or ASA.

An example of this behavior is given below by

revisiting the troubleshooting scenario shown
at the beginning of this paper.

Copyright ©2009 Global Knowledge Training LLC. All rights reserved. 12

 Perimeter
Router

200.200.1.0/24 Outside-PC
DMZ Subnet 150.150.1.20
172.16.1.0/24

ASA/PIX

DMZ-Srv
172.16.1.15
10.10.10.0.2
NAT: 200.200.1.15

Admin-PC
10.10.10.10

The managed device in this example is the Perimeter Router, which will be logged into by the sensor using
telnet. The traces below illustrate the both the translation of the IPS Command and Control interface private IP
address and the occupied terminal line of the router using this translated address.

ASA# show xlate

-- (output omitted) --
Global 200.200.1.52 Local 10.10.10.200

PERIM#show users
Line User Host(s) Idle
Location
* 0 con 0 admin idle 00:00:00
514 vty 0 idle
00:00:05 200.200.1.52

As shown above, the sensor’s IP address of 10.10.10.200 is being translated to 200.200.1.52, and this IP ad-
dress is being used to log in the Perimeter Router terminal line vty0.

Copyright ©2009 Global Knowledge Training LLC. All rights reserved. 13

Summary
By now you should understand troubleshooting the basic proper operation of the Cisco Intrusion Prevention Sys-
tem (IPS). We have intentionally omitted the discussion of auxiliary (although frequently used) monitoring tools
such as IPS Event Viewer, the newer IPS Manager Express, and CiscoSecure MARS. If alerts can be seen on the
CLI console of the sensor, but not by any of these tools, the problem usually lies in the mis-configuration of the
monitoring tool along with the omission of adding the monitoring IP address as an “allowed host” on the IPS.
Once the IPS sensor has been properly installed and the management and sensing interfaces have been initial-
ized, the next task is to ensure that signature alerts are being observed. With the successful viewing of these
alarms, the accompanying challenge frequently becomes tuning the appliance or module to match the alerting
behavior to the environment. Further VPN troubleshooting techniques are addressed in the other two white
papers in this series: Troubleshooting Part I – Connectivity Through ASA or PIX Firewall and Troubleshooting Part
II: Virtual Private Networks.

Learn More
Learn more about how you can improve productivity, enhance efficiency, and sharpen your competitive edge.
Check out the following Global Knowledge courses:
IINS (Introduction to IOS Network Security)
SNRS (Securing Networks with Cisco® Routers and Switches)
SNAF (Securing Networks with ASA Fundamentals)
SNAA (Securing Networks with ASA Advanced)
MARS (Cisco® Monitoring Analysis and Reporting System)
CANAC (Cisco® Appliance for Network Admission Control)

For more information or to register, visit www.globalknowledge.com or call 1-800-COURSES to speak with a
sales representative. Our courses and enhanced, hands-on labs offer practical skills and tips that you can imme-
diately put to use. Our expert instructors draw upon their experiences to help you understand key concepts and
how to apply them to your specific work situation. Choose from our more than 700 courses, delivered through
Classrooms, e-Learning, and On-site sessions, to meet your IT and management training needs.

About the Author

Douglas B. McKillip, P.E., CCIE #1851 is the president and principal consultant of Innovative Integrators Incorpo-
rated, a Delaware Corporation actually based in Delaware. In addition to a BS and MS in Chemical Engineering
from M.I.T., Doug also later obtained an M.S. from the University of Delaware in Computer and Information
Science. After 15 years of experience at DuPont and a brief stint with the original startup company associated
with the Raptor EagleTM Firewall, Doug began his now 15+ year career of teaching and consulting, specializing
in Internet Security with hardware from Cisco Systems® since 1998.

Doug’s hobbies include playing piano at his local church and physical activities such as running, cycling, swim-
ming, golf, skiing, and windsurfing. He and his wife Karen reside in Wilmington, Delaware. Doug can be reached
at innovativeint@mindspring.com.

Copyright ©2009 Global Knowledge Training LLC. All rights reserved. 14