Académique Documents
Professionnel Documents
Culture Documents
Yoshinori Kitatsuji
Katsuyuki Yamazaki
KDDI R&D Laboratories, Inc.
2-1-15 Ohara Kamifukuoka-shi Saitama, Japan
{kitaji, yamazaki}@kddilabs.jp
Abstract
It is getting more difcult to monitor multiple services as
well as to detect and/or to trace Daniel of Service attacks
with only tools showing graphs of the whole IP layer trafc
like MRTG or by checking counters of router interfaces. In
this paper, we discuss the specication of a software-based
real-time measurement tool for ow which consists of multiple capture devices, a manager device and user interface
devices, enabling exible ow denition on demand without
stopping system and working with IPv4 and/or IPv6, while
also enabling high performance. With this discussion, we
propose its architecture, bit-pattern-based ow denition
method and data structure. Then we report on the performance evaluation of a prototype of proposed real-time ow
measurement tools developed on PC-UNIXs and show that
the number of bit-pattern composing ow denitions impact
on the performance. Lastly we show an example of measuring ows in a real world environment and conrm that the
ow extraction is simplied.
1. Introduction
It is getting more difcult to monitor multiple service
classes as well as to detect and/or to trace Daniel of Service
attacks by using only tools showing graphs of the whole
IP layer trafc like Multi Router Trafc Grapher (MRTG)
[1] or by checking counters of router interfaces. It is much
more useful to use a ow visualization tool that measures
ows using any portion from the IP headers up to application data transported by the transport layer when this kind
of problem exists. However, it is expensive to develop hardware equipment measuring ows on high speed links with
such methods because of the complexity of the ow identication processes and with the number of ows detected in
short time on such high speed links.
In this paper, we propose a specication of softwarebased real-time measurement tools of ow which consists
of multiple capturing devices, a manager device and user
interface devices, and propose its architecture, bit-patternbased ow denition method and data structure. Then we
Proceedings of the 2004 International Symposium on Applications and the Internet (SAINT04)
0-7695-2068-5/04 $20.00 2004 IEEE
Ethernet
Switch
Capturing devices
Manager devices
User Interfaces
3.1. Specication
We discuss requirements that capturing and ow identication processes are performed on multiple devices in parallel to bring out the high performance in ow identication
process (Figure 1).
Proceedings of the 2004 International Symposium on Applications and the Internet (SAINT04)
0-7695-2068-5/04 $20.00 2004 IEEE
Manager device
Main
thread process
Statistics is saved in
files periodically
Capturing device
Memory
... ....... ......
Flow identification
thread process
Buffer
Captured Packets
are buffered
Packet buffering
thread process
NIC
Capturing device
Capture Device
Capturing device
Capture Device
Packets are captured at a Capturing device
3.2. Architecture
We propose an architecture of distributed real-time ow
measurement tool as shown in Figure 2 based on the requirements of the previous section.
Firstly, trafc is copied by an Ethernet switch or ber
optic coupler and forwarded to a distribution device. Then
distribution device forwards each packet to one of multiple
capturing devices. we assume that a general-purpose distri-
Proceedings of the 2004 International Symposium on Applications and the Internet (SAINT04)
0-7695-2068-5/04 $20.00 2004 IEEE
bution device is used and that it has a round-robin distribution method, in this paper.
There are 3 thread processes running in capturing devices as described below:
Packet buffering process is deployed to absorb burst
trafc occuring in short time.
Flow identication process identies every packet for
all ow denitions and calculates statistics.
Report and denition update process periodically
makes reports to a manager device and receives message to update ow denitions from a manager device.
The use of PCAP library[8] is one of solutions for capturing processes in a capturing device. A PCAP library can
capture both IPv4 and IPv6 concurrently and make it easier to implement/port a capturing device on/to the variety of
OSs. Additionally, it provides an interface to collect capturing loss, to application programs attempting to detect and
count.
The same ow denitions are advertised to multiple capturing devices from a manager device. The report and denition process controls to derive or remove a packet buffering process and a ow identication process when a capturing device starts. All processes share ow denitions and
statistics constructed in memory space.
There are 4 types of thread processes operating in a manager device as described below.
Main process manages the connections establishment
from capturing devices or user interface devices and
derives the 3 processes below.
Denition advertisement and collection process receives reports from capturing devices and advertises
ow denitions when detecting any difference between registered denitions and denitions in a report.
Save process stores data collected from multiple capture devices periodically.
User interface service process receives ow denitions or sends ow data from/to the user interface devices.
A manger accepts connections from capturing devices
at any time to add and remove capturing devices on demand. A manger device has a mechanism to lter connections based on source address at the same time.
Proceedings of the 2004 International Symposium on Applications and the Internet (SAINT04)
0-7695-2068-5/04 $20.00 2004 IEEE
Generate variety of
flows to a Capturing
device
tree.
Switching Hub
Register Flow
definition when
a capturing device
starts
Switching Hub
Manager Device
Generater
Capturing Device
Xeon 2.8GHz
2GB
73GB
PCI-X (64bit, 133MHz)
2 ports, 10/100Base-TX
RedHat 9 Linux kernel 2.4.20
4.1. Implementation
We measured capturing loss at a capturing device when
sending 500 30K packets per second (pps) respectively
from a trafc generator to a capturing device in Figure 4
conguration. Flow denitions were registered with a manager device through a user interface device and manager
device advertising ow denitions to a capturing device.
A capturing device made a report of every ow identier
counter to a manager device every 10 seconds. All devices
had the same specications as shown in Table 2.
We calculated the average capturing loss from 10 examinations for each packet speed in the following performance
Proceedings of the 2004 International Symposium on Applications and the Internet (SAINT04)
0-7695-2068-5/04 $20.00 2004 IEEE
Number of capt
ure loss per seco
nd
Figure 5. Capturing loss per Number of BitPatterns with Packet Buffering Process
tbi nge
of ra
ze rn
Si tte
pa
Number of capt
ure loss per seco
nd
of rns
er te
mb at
Nu t-p
bi
om a
ted fr
ce
genera ure devi
Capt
(Kbps)
speed ter to a
et
ra
ck
Pa
Gene
c
fi
Traf
om a
ted fr device
genera
ing
(Kbps) a Captur
speed
r to
Packet c Generate
Traffi
Number of capt
ure loss per seco
nd
Number of
capture los
s per second
Figure 6. Capturing Loss per Number of BitPatterns without Packet Buffering Process
tests. Every capturing loss in an examination was calculated
as loss per second during 60-second trafc generation. The
trafc generation was shaped at the user program level to
keep the jitter shorter. However small burst trafc was observed during generation because the nal packet treatment
was controlled by a kernel.
t
bi nge
of ra
ze rn
Si tte
pa
of rns
er te
mb at
Nu t-p
bi
om a
ted fr
vice
genera
(Kbps) a Capture de
speed
Packet Generater to
c
Traffi
om a
ted fr vice
de
genera
bps) a Capture
(K
d
spee
r to
Packet c Generate
Traffi
Proceedings of the 2004 International Symposium on Applications and the Internet (SAINT04)
0-7695-2068-5/04 $20.00 2004 IEEE
1100
Packet size
Traffic speed
96
88
1000
80
72
64
800
56
700
48
600
40
900
32
500
24
400
16
300
8
-8 -7 -6 -5 -4 -3 -2 -1
2 3 4 5 6 7
hour of 2003/June/18
9 10 11 12 13 14 15 16
5. Discussions
5.1. Performance
From the results of Figure 5, it became clear that
the number of bit-patterns signicantly impacts on performance. This implies that the ow denition consisting
of chained bit-patterns proposed in this paper is expected
to prevent drops in capturing performance where multiple
ow denitions are provided and where bit-patterns specifying the same eld comprising different ow denitions
are omitted.
It would be better to modify the order of chained bitpatterns operating as and operations between bit-patterns
while the system is in progress in order to improve the performance. This tuning to reduce average number of pattern
matching per packet depends on the trafc measured at that
moment. The dynamic matching order method is expected
Proceedings of the 2004 International Symposium on Applications and the Internet (SAINT04)
0-7695-2068-5/04 $20.00 2004 IEEE
TCP others
UDP
ICMP
Packet/sec
TCP DST
port = NNTP
TCP DST
port >= 1024
2003/June/24
In the evaluation, we showed that packet buffering process effectively reduce capturing failure, that a capturing
device performs 2K 15K pps where 4 32 bit-patterns
comprising the ow denition are given and that number
of bit-patterns impact signicantly on performance. We
suggested our proposed chained bit-pattern based ow definition is effective for preventing performance failures in
multiple ow denition environments. We also showed the
performance difference between the AVL tree search and
sorted list search as the search method and indicated the criterion at 1024 of range size in selecting one of two searches
where our prototype is used.
Finally, we showed an example of use for our prototype
and were able to conrm that the procedures from ow definition and extracting ows to checking these ows are performed much more easily with a user interface device.
References
[1] Multi Router Trafc Grapher, http://www.mrtg.org/
[2] the Cooperative Association for Internet Data Analysis, http://www.caida.org/
[3] CAIDA, cowd: Trafc Flow Analysis Tool,
http://www.caida.org/tools/measurement/cowd/
[5] Cisco Systems, Inc., NetFlow Service and Applications, White paper, http://www.cisco.com/warp/public/cc/pd/iosw/ioft/nect/tech/napps wp.htm, 2002
[6] Mills, D.L., A. Thyagarajan and B.C. Huffman, Internet timekeeping around the globe, Proc. PTTI Applications and Planning Meeting, Long Beach CA, pp.
365371, December 1997.
[7] J. Quittek, T. Zseby, B. Claise, S. Zander:
Requirements for IP Flow Information Export, Internet Draft (draft-ietf-ipx-reqs-10.txt),
http://www.ietf.org/internet-drafts/draft-ietf-ipxreqs-10.txt, June 2003.
[8] Tcpdump/Pcap, http://www.tcpdump.org/
6. Conclusions
We claried the specication for software-based realtime ow measurement tools which identify ows by using multiple devices, designed and implemented the tools
on PC-UNIXs, and evaluated the performance of a capturing device. We proposed the chained bit-pattern based ow
denition method and data structure for ow identication
and process functions in a capturing device and a manager
device enabling exibly to add or remove capturing devices
while the system is operating.
[9] G. M. Adelson-Velskii and Y. M. Landis: An algorithm for the organization of information, Soviet
Math. Dokl., 3:12591262, 1962.
[10] Asia-Pacic
Advanced
http://www.apan.net/
Network,
Proceedings of the 2004 International Symposium on Applications and the Internet (SAINT04)
0-7695-2068-5/04 $20.00 2004 IEEE