Friday, August 02, 2002

ISP Essentials Supplement

Whitepaper is supplement to the Cisco Press publication The ISP Essentials by Barry Raveendran Greene, and Philip
Smith. Materials can be used with the permission of the authors and Cisco Press. Materials can be used with the permission of
the authors and Cisco Press. Public copies are available at www.cisco.com/public/cons/isp/essentials/ or www.ispbook.com .

Remote Triggering Black Hole Filtering

Supplement to the Existing Section on Black Hole Filtering
Draft 0.2
Send comments, questions, and corrections to Barry Raveendran Greene bgreene@cisco.com

INTRODUCTION
Black hole filtering is a flexible ISP Security tool that will route packets to Null0 (i.e.
black holed). The Cisco ISP Essentials book covers the fundamentals of the single
router based black hole routing technique. It does not cover the remote triggered
black hole routing technique. Remote triggering via iBGP allows ISPs to active a
network wide destination based black hole throughout their network. This technique
is especially useful in some of the new ISP security classification, traceback, and
reaction techniques. This supplement reviews, enhances, and adds to what is
already in the book.

BLACK HOLE ROUTING AS A PACKET FILTER (FORWARDING TO

NULL0)
Forwarding packets to Null 0 is a common way to filter packets to a specific
destination. These are often done by creating specific static host routes and point
them to the pseudo interface Null0. This technique commonly refereed as black
hole routing or black hole filtering. Null0 is a pseudo-interface, which functions
similarly to the null devices available on most operating systems. This interface is
always up and can never forward or receive traffic. While Null0 is a pseudo interface,
within CEF, it is not a valid interface. Hence, whenever a route is pointed to Null0, it
will be dropped via CEFs and dCEFs
The null interface provides an alternative method of filtering traffic. You can avoid the
overhead involved with using access lists by directing undesired network traffic to the
null interface. The following example configures a null interface for IP route
127.0.0.0/16 and the specific host 171.68.10.1 (subnet mask 255.255.255.255):
interface Null0
no icmp unreachables
ip route 127.0.0.0 255.0.0.0 null 0
Cisco Systems, Inc.
170 West Tasman Drive.
San Jose, CA 95134-1706
Phone: +1 408 526-4000
Fax: +1 408 536-4100

Friday, August 02, 2002

ISP Essentials Supplement

ip route 171.68.10.1 255.255.255.255 null 0

The no icmp unreachables command is used to prevent unnecessary ICMP

Unreachable replies whenever traffic is passed to the Null0 interface. This
minimizes the risk of router getting overloaded with hundreds of pending ICMP
Unreachable replies. So it is common practice to use the no ip unreachables
command on the Null0 interface. Yet, they may be cases where you want the router
to respond to with ICMP Unreachables. For these cases, you the ip icmp
unreachable rate-limit command to minimize the risk of a router getting over loaded
with ICMP Unreachable processing. This specific rate-limiting command adjusts the
default of on ICMP Unreachable every 500ms to a value between 1ms to
4294967295 ms.
ip icmp rate-limit unreachable 2000
ip icmp rate-limit unreachable DF 2000

Figure 1 give a graphic example of how this black list filtering technique works.
Black Hole filtering uses the strength of the routers forwarding performance to drop
black listed packets. A router's #1 job is for forward packets - not filtering packets.
The black hole routing technique uses the packet forwarding power to drop all
packets bound for sites on the black list. In the ASIC forwarding world, this black
holing has zero impact in the performance of the router (packets black holed to Null0
are cleared through a register clock). Software forwarding devices have some extra
cycles needed to clear out and black holed packet. If a software-forwarding device is
expected to do a lot of black hole work, consider a black hole shunt interface (see
the section on black hole shunts).
There are two main limitations to with the black hole filtering technique. First, black
hole filtering is L3 only not L4. So access to all L4 services at a give site will be
blocked. If selective L4 filtering is necessary, use extended ACLs. For example, if
you wish to drop all packets to a specific destination, the black hole filtering is
applicable. But, if you wish to drop all telnet packets to a destination, then black hole
filtering is no applicable and a extended ACL is the optimum mitigation tool.
Extended ACLs offer the fine L4 granularity needed to filter at the application level.
Second, it is hard to bypass or provide exceptions with the black hole filtering
technique. Any organization that wishes to by-pass the black list must actually find a
way to by-pass the filtering router's forward table. Compensation for either limitation
are not trivial tasks. Yet. With due consideration and planning, options are available
for both.

Cisco Systems, Inc.

170 West Tasman Drive.
San Jose, CA 95134-1706
Phone: +1 408 526-4000
Fax: +1 408 536-4100

Friday, August 02, 2002

ISP Essentials Supplement

Figure 1 - Using static host routes to null 0 for black list filtering

REMOTE TRIGGERED BLACK HOLE FILTERING

Black Hole Filtering on a single router has been around the industry since the last
1980s. It is a useful tool on a single router. But, how do you use this tool when you
have a network of hundreds of routers? How do you log into hundreds of routers on
the edge of a network and configure a black hole filter? The answer is in you dont.
ISPs engineers who respond to a security incident needs to think of their key
strength routing. ISPs engineers know how to route traffic putting the traffic
where they want it to flow through their network. Remote Triggered Black Hole
Filtering uses that routing strength to trigger all the routers in the network with a
routing update. The routing update sent via iBGP by a trigger router actives a
pre-configured static route to activate a black hole for the destination address.
Lets use an example to illustrate the concept and strength of this technique. Figure
2 illustrates an ISPs customer under attack by a DDOS. The DDOS is coming in
from all the entry points of the ISPs network. These entry points can number from a
few to thousands depending on the size of the ISP. DDOS traffic far exceeds the
customers link, so the circuit saturates, causing either DOS Flapping1 or co-lateral
damage inside the POP. This collateral threats other customers, the POP, and that

DOS Flapping is a form of co-lateral damage that comes when the circuit under attack goes into
congestion collapse. The DOS Flap happens when the IGP route locking the iBGP advertisement for
the customer drops with the saturated circuit. The more specific for that customer is removed, flapping
the DOS flow to the next best path. That could be a router in the POP or somewhere else in the
network. Note: Sink Holes might be a way to shunt DOS Flaps so they do not cause collateral
damage in the POP.
Cisco Systems, Inc.
170 West Tasman Drive.
San Jose, CA 95134-1706
Phone: +1 408 526-4000
Fax: +1 408 536-4100

Friday, August 02, 2002

ISP Essentials Supplement

section of the ISPs network. An immediate reaction is necessary to shift the packet
drops from the customers circuit and collateral routers to the edge of the network.
Peer A

IXP-W
A

Peer B

IXP-E
Upstream
A

Upstream
A

Upstream
B

Upstream
B

Target

F POP

Target is taken
out

NOC

Figure 2 - Customer Under Attack by a DDOS

Remote Triggered Black Hole filtering is used to push the packet drops off the
customer/POP routers and shift them to the edge of the network. Figure 3 shows
how an ISP uses a trigger router in the NOC to send an iBGP advertisement. This
iBGP advertisement has the prefix of the customer under attack with metric attached
to insure it becomes the preferred path. This iBGP trigger advertisement goes to all
the iBGP specking routers in the ISPs network. These routers all have an unused
prefix that points to Null 0. The iBGP trigger advertisement has its next-hop equal
to this Null0ed prefix. When the iBGP trigger advertisement reaches the router, it
gets glued to the static, activating the Null0 black hole, and having all traffic to the
customers prefix get dropped on the edge of the ISPs network.
The key benefit in this situation is that dropping on the edge of the network mitigates
the DDOSs aggregated traffic load. This now gives the ISP and the customer time to
work the attack with out the worries of collateral damage to other customers.

Cisco Systems, Inc.

170 West Tasman Drive.
San Jose, CA 95134-1706
Phone: +1 408 526-4000
Fax: +1 408 536-4100

Friday, August 02, 2002

IXP-W
A

ISP Essentials Supplement

Peer A
Peer B

IXP-E
Upstream
A

Upstream
A

Upstream
B

Target

F POP

NOC

Upstream
B

iBGP
Advertises
List of
Black Holed
Prefixes

Figure 3 - Trigger Router Activates the Black Hole Throughout the Network

DOES REMOTE TRIGGERED BLACK HOLE FILTERING REALLY

WORK?
Some of the largest ISPs on the planet use remote-triggered black hole filtering.
WorldCom/UUNET is the best example. As seen in the NANOG 23 Tutorial - ISP
Security - Real World Techniques (see http://www.nanog.org/mtg-0110/greene.html),
Christopher L. Morrow and Brian W. Gemberling demonstrated how they use remote
triggered blackhole filtering together with sink holes and backscatter to create a
innovative tool to rapidly traceback attacks to the edge routers on their network
(usually takes between 5 to 15 minutes across AS numbers 701, 702, and 703). So
the response for those who doubt the technique can be assured that it does work. If
UUNET can integrate this technique as one of their core DOS/DDOS response tools,
then there should be little problem for others to implement the technique.

REMOTE TRIGGERING SAFETY MEASURES

Remote Triggering via iBGP requires the ISP to take some safety measures to
insure the iBGP trigger advertisement does not leak out and affect other networks.
There are several ways this can be done. Appling the principle of Murphys Law of
Networking, it is recommended that an ISP implement several if not all of these
safety measures.
Cisco Systems, Inc.
170 West Tasman Drive.
San Jose, CA 95134-1706
Phone: +1 408 526-4000
Fax: +1 408 536-4100

Friday, August 02, 2002

ISP Essentials Supplement

No-export BGP community. The no-export community in BGP is a wellknown value that most routers recognize by default. It should when working
properly keep the prefix within the ISP (i.e. no advertisements to peers).

Extra Community that filters. The ISP can add a community that does the
same as the no-export community. A BGP community filter will be used on
with the ISPs peers to mark which communities are exported. This step helps
prevent a leak by someone who is cleaning up the excess communities in the
prefix inadvertently filtering the no-export community.

Lower Boundary on the Egress Prefix Filter. ISPs can place a lower
boundary on the prefixes sent to their peers. For example, ISPs can block all
prefixes less than /24. This would filter any iBGP trigger advertisement
between /25 and /32 which is a normal range of addresses blocks allocated
to customers.

PREPARING THE NETWORK FOR REMOTE TRIGGERED BLACK

HOLE FILTERING
It is imperative that ISPs prepare for remote triggered black hole filtering, practice the
technique, and have everything ready long before using it to mitigate an attack.
Fortunately, all the preparation steps involve non-intrusive configurations that have
no impact on the operation of the network.

Step 1- Configure the Static Route to Null0 on All the Routers

The first of these preparation steps is the configuration of a static route on each of
the routers that will be triggered. This is a prefix that will never be used in the
network. It can be a block of addresses allocated from the RIR allocations. It can be
a RFC 1918 prefix. The authors favorite is to use the Test-Net: 192.0.2.0/24. TestNet was a IANA allocation made for people to do documentation. The idea was for
documentation to use a block of addresses that would never get used. That way
customers who copy the documentation will not mess up someone elses network.
Hence, Test-Net is one of the IANA Designated Special Use Addresses (DUSA) that
should never appear on the Internet making it a great choice for the static route
for remote-triggered Black Hole Filtering.
ip route 192.0.2.0 255.255.255.0 Null0

Step 2 Prepare the Trigger Router

Cisco Systems, Inc.
170 West Tasman Drive.
San Jose, CA 95134-1706
Phone: +1 408 526-4000
Fax: +1 408 536-4100

Friday, August 02, 2002

ISP Essentials Supplement

The trigger router does not have to be a big router. A Cisco 26XX or 36XX router
configured as an iBGP route reflector client and accepting no routes works very well
as a trigger router. In fact, the trigger router does not have to be a dedicated router.
A production router can be used. For this example, we will be using a dedicated
trigger router.
On the router, the iBGP is configured to redistribute static routes. That way the
trigger is an engineer or tool adding and removing static routes. A route-map is
used to match the static tag and set all the metrics for the iBGP advertisement. That
way all triggering is consistent and done the same way each time.
router bgp 109
.
redistribute static route-map static-to-bgp
.
!
route-map static-to-bgp permit 10
match tag 66
set ip next-hop 192.0.2.1
set local-preference 50
set community no-export 600:000
set origin igp
!
Route-map static-to-bgp permit 20

In the above example, we match a static tag of 66. If matched, we set the iBGP nexthop to the Test-Net (pre-configured on the routers to Null0), set the local preference
to 50 (to override the original customer advertisement), set the BGP community to
no-export with a safety community of 600:000 (which blocks advertisement, and
finally set the origin to igp. This sets up the trigger router to be ready for the time
when the ISP needs for rapid reaction.

Step 3 - Activation
The ISP adds a static route with a tag of 66 to activate the remote-triggered black
hole. In this example, well use 171.68.1.1 as a the address under attack. So we add
this static with the tag of 66:
ip route 171.68.1.1 255.255.255.255 Null0 Tag 66

The trigger router will then send a advertisement to all the iBGP speaking routers in
the network (see Figure 3). When the iBGP advertisement is received, the BGP RIB
sees the local preference of 50 and selects this new path as the best path. The
recursive look-up passes since there is a static route to this new paths next-hop (i.e.
the Test-Net). This iBGP best path is passed from the BGP RIB to the routers FIB.
The FIB sees the prefix, the next-hop of Test-Net, and Test-Nets next-hop of Null0.
It then glues them together (depending on the FIB technology used) resulting
171.68.1.1 now having a next-hop of Null0. This is visually illustrated in Figure 4.
Cisco Systems, Inc.
170 West Tasman Drive.
San Jose, CA 95134-1706
Phone: +1 408 526-4000
Fax: +1 408 536-4100

Friday, August 02, 2002

ISP Essentials Supplement

BGP Sent 171.68.1.1 Next-Hop = 192.0.2.1

Static Route in Edge Router 192.0.2.1 = Null0

171.68.1.1 = 192.0.2.1 = Null0

Next hop of 171.68.1.1 is now equal to

Null0
Figure 4 - How the Router takes the iBGP Trigger and Activates the Black Hole

One of the key advantages of remote triggered black hole is the number of prefixes
that can be filtered. The limit is the size of the FIB routers in the network can carry.
This would mean thousands of black holed prefixes being added. It is just a factor of
adding more static routes to the trigger router. Principles of aggregation can be used,
but mindfulness needs to be applied to make sure the iBGP trigger advertisement is
equal to or more specific to the original customer advertisement.

Step 4 Removing Trigger Advertisement

The trigger advertisement will need to be removed when the attack is over or the ISP
wishes to move to a different mitigation technique. Removing the static route does
this. The trigger router will then send out a iBGP withdrawal to all its BGP peers,
which in then will withdraw the route from the BGP RIB, which then pulls the route
from the routes FIB. This clears the path for the routers BGP RIB to select the
original customer advertisement, placing that prefix as the best path, and allowing
the FIB to resume normal forwarding to the customers network.

LIMITATION OF REMOTE TRIGGERED BLACK HOLE FILTERING

Each security tool has its strengths and limitations. The key limitation of remotetriggered black hole filtering is the effect of an un-intended consequence. An
example of an un-intended consequence is having a DDOS attack specifically
Cisco Systems, Inc.
170 West Tasman Drive.
San Jose, CA 95134-1706
Phone: +1 408 526-4000
Fax: +1 408 536-4100

Friday, August 02, 2002

ISP Essentials Supplement

designed to force an ISP to black hole a customer. The attack would be set up to
cause co-lateral damage to adjacent customers. The collateral damage forces the
ISP to react with a remote-triggered black hole taking out the path to the customer.
The problem is that the black hole is exactly what the attacked wanted removing
the customers ability to receive traffic.
This limitation is one of the reasons ISPs need a suite of mitigation tools. Focus on
any single DOS/DDOS mitigation tool limit an ISPs options and make them
vulnerable to a manipulating attack profile where the attack manipulates the ISP to
react to the benefit of the attacker.

REMOTE-TRIGGERED BLACK LIST FILTERING

Black List filtering refers to organizations attempting to place barriers to information
on the Internet. The Internet - by it's nature, culture, and heritage - resist any barrier
to communication. In fact, the Internet is the ultimate of internetworking where
everything and everyone is interconnected through transparent end-to-end
connectivity. Yet, because of the social and political forces the Internet brings with it,
there are organizations and government who wish to block and/or protect
customers/citizens from access certain sites on the Internet. Pornographic, Political,
and On-Line Gambling are the top three topics that organizations and governments
wish to include in a black list filter.

_____________________________________________

Penn. Law Makes ISPs Liable If Child Porn Sites Not

Blocked From Users
Harrisburg, Pa. -- The state of Pennsylvania has passed a law that will make it
a crime if ISPs operating in the state do not block their users access to child
pornography websites, the Associated Press reported. Under the law, which
goes into effect in April, the state attorney general's office will have the
responsibility of informing ISPs of which sites to block. However, since such
sites often are targets of law enforcement, they frequently change their Web
addresses and operate overseas. ISPs violating the law will be fined $5,000 for
the first offense, $20,000 for the second and $30,000 and up to seven years
imprisonment for any further violations.

http://www.msnbc.com/news/725968.asp?0dm=C14OT
_____________________________________________
Remote-Triggered Black Hole filtering is one way to accomplish Black List filtering. If
an ISP has a legal requirement to block access to specific sites, destination black
Cisco Systems, Inc.
170 West Tasman Drive.
San Jose, CA 95134-1706
Phone: +1 408 526-4000
Fax: +1 408 536-4100

Friday, August 02, 2002

ISP Essentials Supplement

holes will accomplish that objective. The core issue is the operational overhead of
maintaining that list. Remote-triggered black hole filtering helps minimize the
operational overhead by having one router in one location holding the master list (i.e.
the trigger router). This allows for frequent and rapid change to the black list. This
change feature is critical, since as mentioned earlier, the Internet resist attempts to
block the flow of information. Once a site is black listed, that site will physically or
logically move. A simple change of IP addresses could be all that it would take.

ACKNOWLEDGEMENTS
While I do not remember who taught me this technique, the first time I personally
used it was back in 1991 on a couple of US Military networks I was helping to
operate. Since that time, this technique seemed to be forgotten explaining why
many people did not understand how uRPF Loose Check and Source Based
Remote-Triggered Black Hole Filtering will work (see the separate paper on that
topic). It was not until Christopher L. Morrow chris@UU.NET, Brian W. Gemberling
brian@UU.NET, and UUNETs NetSec Team demonstrated to the world that
Remote-Triggered Black Hole filtering as part of their Backscatter Traceback
technique that people woke up to the power of this mitigation technique. So a lot of
credit goes to UUNET for showing that this technique works. As some say, if it can
work at UUNET, then 99% of the deployment issues have been covered. J

Cisco Systems, Inc.

170 West Tasman Drive.
San Jose, CA 95134-1706
Phone: +1 408 526-4000
Fax: +1 408 536-4100

10