Pierky's Blog

mostly a system and network engineer's repository

Home

About

Tools & Download

PGP

Disclaimer

Twitter

NetFlow: how to install and configure flow-tools and FlowViewer on a

fresh Debian setup
Posted on 6 March 2010 by pierky

21 Comments

RSS

About me

My .tel

Leave a comment

My Google+ profile

NetFlow is a very useful tool/protocol to monitor network traffics patterns. Many tools have been developed

My LinkedIn profile

to collect and analyze NetFlow data, here I chose flow-tools and FlowViewer packages, and I would like
to show how to get them work on a fresh Debian 5.0 (Lenny) setup.

SNMP Monitoring Software

helpsystems.com/intermapper

SNMP
Management
Software

Be in the know! Hear about router & server problems with InterMapper.

Components used in the solution

There are two main packages involved in the solution: flow-tools, used to acquire and collect NetFlow data

helpsystems.com/intermapper

generated by our routers, and FlowViewer, used to process, view and paint them on nice graphs.
The flow-tools package is built up by a lot of components, many of them are transparently used by
FlowViewer; here Ill focus on the flow-capture program, the one which acquires and collect the data.
The FlowViewer package is splitted up on 3 CGIs: FlowViewer, FlowGrapher and FlowTracker. They

Categories

can be used through a web-server and they let us analyze data collected by flow-capture. The package
contains 2 programs also, FlowTracker_Collector and FlowTracker_Grapher, which run periodically and
build MRTG-like graphs, storing data in RRD databases.

Italy focused
Networking
Security

Acquiring data

Networking Labs
Systems Administration

First off, we need to acquire NetFlow data generated by our routers; flow-tools is the package we need:

Uncategorized

apt-get install flow-tools

Tags

Lets edit the main configuration file, flow-capture.conf, where we tell flow-capture what we want to
802 DOT1Q 802.1q ACS

acquire and how we want to store it:

1
2

cd /etc/flow-tools
nano flow-capture.conf

flow-tools FlowViewer Fun GNS

flow-capture: each line is used to run an instance of the program.

example:

# MYSECONDROUTER
-V 5 -E 5G -N 3 -w /var/flows/MYSECONDROUTER 0.0.0.0/192.168.0.2/3002

So, I acquire data from MYROUTER, which sends NetFlow version 5 data from 192.168.0.1; this flowcapture instance will be listening on port 3001 of every local IP address (0.0.0.0) and it will store data on

GNS3

ISP LBE LinkedIn MPLS Multicast

NAT NetFlow NMS OpenACS PBR

You can execute man flow-capture to view all the arguments it accepts; here I used the following

# MYROUTER
-V 5 -E 5G -N 3 -w /var/flows/MYROUTER 0.0.0.0/192.168.0.1/3001

EIGRP

GNS3 Lab GNS3 Lab and Exercise

GNS3 Topology HowTo HSRP IPv6

Its configuration file is pretty simple; its built up by many lines containing the command line arguments of

1
2
3
4
5

BGP BSCI CCNP

Cisco CPE CWMP DNS DNSSEC

Provider QoS Radius Remote Triggered

Black Holing RIPE RTBH Script Security

SNMP TR-069 uRPF Zabbix

Follow me on Twitter

Tweets

Follow

the /var/flows/MYROUTER directory, with a nesting level of type 3, that is directories like
/var/flows/MYROUTER/YYYY/YYYY-MM/YYYY-MM-DD/. It will keep files up to a maximum of 5 GB.
This may be a sample configuration for MYROUTER (Cisco), where 192.168.0.9 is the IP address of our
NetFlow box:

ip flow-export source FastEthernet0/0.1

Pier Carlo Chiodi

@pierky
pic.twitter.com/uNTTZIPx1H

15 Jul

ip flow-export version 5
ip flow-export destination 192.168.0.9 3001
interface FastEthernet0/0
description LAN facing
no ip address
interface FastEthernet0/0.1
Stefano Quintarelli

encapsulation dot1Q 1 native

Tweet to @pierky

ip address 192.168.0.1 255.255.255.0

Search

interface FastEthernet0/1
description WAN facing
ip address 10.0.0.1 255.0.0.0
ip route-cache flow

So on for MYSECONDROUTER
We just have to build the destination directories and then run the program:
1
2
3
4

mkdir -p /var/flows/MYROUTER
mkdir -p /var/flows/MYSECONDROUTER

/etc/init.d/flow-capture start

The tool is now working:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

ls -l -R /var/flows/MYROUTER/
/var/flows/MYROUTER/:
totale 4
drwxr-xr-x 3 root root 4096 5 mar 10:11 2010
/var/flows/MYROUTER/2010:
totale 4
drwxr-xr-x 3 root root 4096

5 mar 10:11 2010-03

/var/flows/MYROUTER/2010/2010-03:
totale 4
drwxr-xr-x 2 root root 4096 5 mar 10:11 2010-03-05
/var/flows/MYROUTER/2010/2010-03/2010-03-05:
totale 4
-rw-r--r-- 1 root root 92 5 mar 10:11 ft-v05.2010-03-05.101125+0100

FlowViewer
Once we are collecting data using flow-tools we want to analyze them, so we need to get FlowViewer up
and running!

Requirements
As we can see from the FlowViewer web site we have to satisfy some requirements in order to run it:
- a web server with CGI support;
- Perl 5.0 or later;
- FlowTools;
- GD and GD:Graph;
- RRDTool.
Lets start installing them:
1
2
3
4
5

apt-get install apache2

apt-get install libgd-graph-perl

apt-get install rrdtool

As well see later in the FlowViewer configuration, it uses also another utility to resolve IP addresses in host

names: dig. We have to install the dnsutils Debian package in order to have it:
1

apt-get install dnsutils

Installation and configuration

Download and untar the package:
1
2
3

cd /usr/local/src
wget http://ensight.eos.nasa.gov/FlowViewer/FlowViewer_3.3.1.tar
tar -xf FlowViewer_3.3.1.tar

Now that all requiremets are met and FlowViewer is on the disk, lets start configuring it!
In order to get FlowViewer up and running we have to edit its configuration file and build some directories
it needs. The user guide provided by the author is very complete, you can find there any information you
need. The web sites FAQ section is very useful too. Here Ill just provide a basic configuration and layout.
In this sample configuration I use the Apaches default web site as starting point, so I have:
- / (the root) on /var/www/
- /cgi-bin/ on /usr/lib/cgi-bin/
Lets move FlowViewer into the cgi-bin directory
1

mv FlowViewer_3.3.1 /usr/lib/cgi-bin/

and edit the configuration file:

1
2

cd /usr/lib/cgi-bin/FlowViewer_3.3.1
nano FlowViewer_Configuration.pm

There are a bit of parameters to change, but remember: the user guide is your friend.
Here is the diff of my file against the original:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

?
diff -y --suppress-common-lines -W 250 FlowViewer_Configuration.pm FlowViewer_Configuration.pm.ORIG
$FlowViewer_server
= "192.168.0.9";
$FlowViewer_service
= "http";
$reports_directory
= "/var/www/FlowViewer";
$reports_short
= "/FlowViewer";
$graphs_directory
= "/var/www/FlowGrapher";
$graphs_short
= "/FlowGrapher";
$tracker_directory
= "/var/www/FlowTracker";
$tracker_short
= "/FlowTracker";
$cgi_bin_directory
= "/usr/lib/cgi-bin/FlowViewer_3.3.1";
$work_directory
= "/usr/lib/cgi-bin/FlowViewer_3.3.1/Flow_Working";
$save_directory
= "/var/www/FlowViewer_Saves";
$names_directory
= "/usr/lib/cgi-bin/FlowViewer_3.3.1";
$filter_directory
= "/usr/lib/cgi-bin/FlowViewer_3.3.1/FlowTracker_Files/FlowTracker_Fil
$rrdtool_directory
= "/usr/lib/cgi-bin/FlowViewer_3.3.1/FlowTracker_Files/FlowTracker_RRD
$flow_data_directory
= "/var/flows";
$exporter_directory
= "/var/flows/all_routers";
$rrdtool_bin_directory
= "/usr/bin";
$trackings_title
= "FlowViewer Saves";
$user_hyperlink
= "/FlowViewer_Saves";
@devices
= ("MYROUTER","MYSECONDROUTER");
$log_directory
= "/usr/lib/cgi-bin/FlowViewer_3.3.1";

And here is every directory or short parameter:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

?
cat FlowViewer_Configuration.pm | grep "directory|short"
$reports_directory
= "/var/www/FlowViewer";
$reports_short
= "/FlowViewer";
$graphs_directory
= "/var/www/FlowGrapher";
$graphs_short
= "/FlowGrapher";
$tracker_directory
= "/var/www/FlowTracker";
$tracker_short
= "/FlowTracker";
$cgi_bin_directory
= "/usr/lib/cgi-bin/FlowViewer_3.3.1";
$cgi_bin_short
= "/cgi-bin/FlowViewer_3.3.1";
$work_directory
= "/usr/lib/cgi-bin/FlowViewer_3.3.1/Flow_Working";
$work_short
= "/cgi-bin/FlowViewer_3.3.1/Flow_Working";
$save_directory
= "/var/www/FlowViewer_Saves";
$save_short
= "/FlowViewer_Saves";
$names_directory
= "/usr/lib/cgi-bin/FlowViewer_3.3.1";
$filter_directory
= "/usr/lib/cgi-bin/FlowViewer_3.3.1/FlowTracker_Files/FlowTracker_Fil
$rrdtool_directory
= "/usr/lib/cgi-bin/FlowViewer_3.3.1/FlowTracker_Files/FlowTracker_RRD
$flow_data_directory
= "/var/flows";
$exporter_directory
= "/htp/flows/all_routers";
$flow_bin_directory
= "/usr/bin";
$rrdtool_bin_directory
= "/usr/bin";
$log_directory
= "/usr/lib/cgi-bin/FlowViewer_3.3.1";
$log_collector_short= "Y";
$log_grapher_short = "Y";

Apart from the directories, please note the $FlowViewer_server and $FlowViewer_service parameters, and
the @devices array, containing the comma-separated list of routers we already configured in flowcaptures.
Now, we have to build the directories used by FlowViewer and, of course, we have to set the needed
permissions on them.
Here they are:
1
2
3
4

mkdir -p /var/www/FlowViewer
chmod -R a=rwx /var/www/FlowViewer
...
do for every directory which needs to be created and set the rights permissions

It just remains to copy some images into the proper directories

1
2
3

cp Generic_Logo.jpg /var/www/FlowViewer/
cp FlowViewer_Save.png /var/www/FlowViewer/
cp FlowViewer.png /var/www/FlowViewer_Saves/

and we are ready to use our NetFlow solution!! Simply point your browser at
http://your_server_IP_address/cgi-bin/FlowViewer_3.3.1/FlowViewer.cgi
Is that all? No, it isnt!

FlowTracker Grapher and Collector

FlowViewer is not just a CGIs collection, it also includes two programs intended to be ran continously on
background: they are FlowTracker_Grapher and FlowTracker_Collector.
Once you defined some trackings in FlowTracker they constantly grab data from flow-capture files, store
them in RRD databases and build graphs, like MRTG does with SNMP data. They also send you a warning
when traffic goes over the thresholds you defined.
To start them up automatically I edited the flowcap script included within FlowViewer package
(/usr/lib/cgi-bin/FlowViewer_3.3.1/flowcap); here it is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73

?
#!/bin/sh
#
# FlowTracker:
Starts all processes concerning FlowTracker
#
# description: This script starts up the the FlowTracker tools (Collector and Grapher)
#
# processname: There is not a single process associated with these
#
actions, rather there are multiple processes. This
#
script takes care of all of them.
#
# can be restarted by using the following command:
#
# sudo /etc/init.d/FlowTracker restart

RETVAL=0
start() {
echo -n $"Starting FlowTracker processes: "
echo -n "FlowTracker_Collector "
cd /usr/lib/cgi-bin/FlowViewer_3.3.1
./FlowTracker_Collector &> /dev/null &
RETVAL=$?
echo -n "FlowTracker_Grapher "
cd /usr/lib/cgi-bin/FlowViewer_3.3.1
./FlowTracker_Grapher &> /dev/null &
RETVAL=$?
echo ""
}
stop() {
echo -n $"Stopping FlowTracker processes: "
RETVAL=0
for p in `pidof perl`
do
ps $p | grep FlowTracker_Collector > /dev/null
if [ $? -eq 0 ]; then
echo -n "FlowTracker_Collector "
kill $p
fi
ps $p | grep FlowTracker_Grapher > /dev/null
if [ $? -eq 0 ]; then
echo -n "FlowTracker_Grapher "
kill $p
fi
done
}

echo ""

# See how we were called.

case "$1" in
start)
start
;;
stop)
stop
;;
restart)
stop
start
;;
*)
echo $"Usage: $0 {start|stop|restart}"
;;
esac
exit $RETVAL

Then I moved it in the /etc/init.d directory as FlowTracker, made it executable and scheduled it to be
executed at startup:
1
2
3

mv flowcap /etc/init.d/FlowTracker
chmod a+x /etc/init.d/FlowTracker
update-rc.d FlowTracker defaults 30

/etc/init.d/FlowTracker

Finally:
1

Housekeeping
Just to clean some files now and then, add an entry in your crontab file pointing to a cleanup script, like

the following one:

1
2

?
cd /usr/lib/cgi-bin/FlowViewer_3.3.1/
/usr/lib/cgi-bin/FlowViewer_3.3.1/FlowViewer_CleanFiles &> /usr/lib/cgi-bin/FlowViewer_3

Give it execution permissions:

1

chmod a+x /usr/lib/cgi-bin/FlowViewer_3.3.1/cleanup

Then put it in your crontab:

# m h dom mon dow user

5

command
/usr/lib/cgi-bin/FlowViewer_3.3.1/cleanup

Everything is done! Enjoy exploring your network traffic and stay tuned for more NetFlow posts!

References
Cisco.com: Cisco IOS NetFlow
Wikipedia: Netflow
Flow-tools: http://www.splintered.net/sw/flow-tools/
FlowViewer: FlowViewer Web Site and F.A.Q.
Switch.ch: List of NetFlow related software

Categories: Networking, Security, Systems Administration | Tags: Cisco, flow-tools, FlowViewer, HowTo, ISP,
LinkedIn, NetFlow, NMS, Provider
NetFlow: weird TCP flags in FlowViewer and flow-print?
Working configuration for Telecom Italia 4 Mbps SHDSL 4-wire bonding

21 Comments
ua0ljj
27 October 2010 at 05:01
Using FlowViewer_3.3.1
I doing:
cat /usr/lib/cgi-bin/FlowViewer_3.3.1/cleanup

Where it is script?

pierky
27 October 2010 at 09:40
Hi,
the Housekeeping paragraph was a bit confusing, I changed it.
The cleanup script is not included in FlowViewer, I made it and there I show how I did.
So, if you want, you have to edit that file and schedule it on your crontab.
Bye
Pierky

megezo
24 November 2010 at 20:03
Excellent tutorial
I have an important question I think, about how flow-capture manages disk space.
What happens when the total size of netflow files reaches the configured limit of 5GB ?
Will flow-capture stop recording ?
How do you face this problem ?

pierky
25 November 2010 at 11:12
Hi Megezo,
the man page says:

-E expire_size Retain the maximum number of files so that the total storage is less than expire_size. The letters
b,K,M,G can be used as multipliers, ie 16 Megabytes is 16M. Default to 0 (do not expire).
I never used it in a production environment, so I have not a personal experience, but I guess it clears old flows and
keeps capturing new data.
Pierky

Megezo
29 November 2010 at 12:42
Hello Pierky,
Indeed it appears that flow-capture clears old files, although the total size of netflow files exceeds a little bit the
Expiration parameter.
Heres my test:
1) Set the Expiration parameter to some value:
vim /etc/flow-tools/flow-capture.conf
# Pierkys blog configuration
-V 5 -E 17500K -N 3 -w /var/flows/MYROUTER 0.0.0.0/192.168.43.1/5502
2) Display subdirectories total size:
netflow-collector-2:~# date
dimanche 28 novembre 2010, 11:08:34 (UTC+0100)
netflow-collector-2:~#
netflow-collector-2:~# du -h /var/flows/MYROUTER/2010/2010-11
840K /var/flows/MYROUTER/2010/2010-11/2010-11-28
4,9M /var/flows/MYROUTER/2010/2010-11/2010-11-26
4,3M /var/flows/MYROUTER/2010/2010-11/2010-11-25
6,7M /var/flows/MYROUTER/2010/2010-11/2010-11-27
17M /var/flows/MYROUTER/2010/2010-11
And the same command, a day later. Note that the oldest directory (2010-11-25) has shrinked, and the total size
exceeds slightly the Expiration limit:

netflow-collector-2:~# du -h /var/flows/MYROUTER/2010/2010-11
6,7M /var/flows/MYROUTER/2010/2010-11/2010-11-28
3,2M /var/flows/MYROUTER/2010/2010-11/2010-11-26
4,0K /var/flows/MYROUTER/2010/2010-11/2010-11-25
6,7M /var/flows/MYROUTER/2010/2010-11/2010-11-27
1,8M /var/flows/MYROUTER/2010/2010-11/2010-11-29
19M /var/flows/MYROUTER/2010/2010-11

pierky
29 November 2010 at 20:20
Hi Megezo,
many thanks for sharing your experience on the blog!
Pierky

Megezo
30 November 2010 at 10:50
Well, thank you for this excellent blog
Megezo

dano
13 December 2010 at 11:41
can you setup NetFlowViewer on CentOS-5.5 ? thank before

pierky
13 December 2010 at 12:38
Hi,
I never installed it on CentOS but I think you can do it, using flow-tools and FlowViewer source code.
Bye

dano
14 December 2010 at 03:49
I already try to do it but when i install flow-tools after i start the service i saw the log file unlink (/var/run/flowcaptuer.pid.8818) Permission denied.
I try to fix this problem but can not do you any idea ?
Thank for your reply.

pierky
14 December 2010 at 11:19
Sorry man, no ideas here. Try to give full permissions on files and directories to the user flow-tools runs on.

Hook
30 March 2011 at 19:03
Be careful when changing directory permissions. Your code will include /usr/bin (since its in the config file) which you
dont want to modify.

cat FlowViewer_Configuration.pm | grep directory | awk -F " '{print $2}'

Thank you for documenting the FlowViewer configuration.

Lay
25 August 2011 at 08:41
Hi Pierky,
I am installing FlowViewer from http://ensight.eos.nasa.gov/FlowViewer/ on Ubuntu 64-bit edition and I can now access
FlowViewer via my browser http://10.6.192.97/cgi-bin/FlowViewer_3.4/FlowViewer.cgi but it doesnt show any outputs.

Could you please help me advise the possible cause of why FlowViewer cant seem to read flow data while Flow Capture
seems to be receiving flows from the router?
setup@ubuntu:~$ ls -l -R /var/flows/MYROUTER/
/var/flows/MYROUTER/:
total 4
drwxrwxrwx 3 root root 4096 2011-08-24 20:17 2011
/var/flows/MYROUTER/2011:
total 4
drwxrwxrwx 3 root root 4096 2011-08-24 20:17 2011-08
/var/flows/MYROUTER/2011/2011-08:
total 4
drwxrwxrwx 2 root root 4096 2011-08-24 23:14 2011-08-24
/var/flows/MYROUTER/2011/2011-08/2011-08-24:
total 18700
-rwxrwxrwx 1 root root 1566957 2011-08-24 20:30 ft-v05.2011-08-24.201715-0700
-rwxrwxrwx 1 root root 1904374 2011-08-24 20:45 ft-v05.2011-08-24.203000-0700
-rwxrwxrwx 1 root root 1810963 2011-08-24 21:00 ft-v05.2011-08-24.204500-0700
-rw-rr 1 root root 1649026 2011-08-24 21:15 ft-v05.2011-08-24.210000-0700
-rw-rr 1 root root 1493551 2011-08-24 21:30 ft-v05.2011-08-24.211500-0700
-rw-rr 1 root root 1517829 2011-08-24 21:45 ft-v05.2011-08-24.213000-0700
-rw-rr 1 root root 1530559 2011-08-24 22:00 ft-v05.2011-08-24.214500-0700
-rw-rr 1 root root 1557358 2011-08-24 22:15 ft-v05.2011-08-24.220000-0700
-rw-rr 1 root root 1588123 2011-08-24 22:30 ft-v05.2011-08-24.221500-0700
-rw-rr 1 root root 1580850 2011-08-24 22:45 ft-v05.2011-08-24.223000-0700
-rw-rr 1 root root 1641061 2011-08-24 23:00 ft-v05.2011-08-24.224500-0700
-rw-rr 1 root root 92738 2011-08-24 23:00 ft-v05.2011-08-24.230000-0700
-rw-rr 1 root root 1162143 2011-08-24 23:14 ft-v05.2011-08-24.230303-0700
-rw-rr 1 root root 16468 2011-08-24 23:14 tmp-v05.2011-08-24.231405-0700
setup@ubuntu:~$
Thank you very much in advance for your kind assistance.
Regards,
Lay

Yogesh Sathe
18 November 2011 at 15:32
Hi,
I find this tool useful to collect data from brach routers. (all routers are Cisco routers)
But for some routers, I am getting the data in /var/flows/ in branch folders directory when seen through ls- l R
command but actually it dont show data in flowviewer tool even after selecting appropriate options I am bigineer in
linux and started with Ubuntu desktop 10.04 lucid.
For other routers I am able to see the netflow version 5 data.
Please advice.

Vaquzanf
4 May 2012 at 16:37
Do you know the address? http://uorysisory.de.tl bd company nymphets I know some working girls would like doing
something like this instead of fucking. Personally, Id rather get fucked and move on to the next.

Luca Maranzano
8 April 2013 at 16:37
Please Please Please read CAREFULLY this command:
for d in `cat FlowViewer_Configuration.pm | grep directory | awk -F {print $2}`; do chmod -R a=rwx $d/; done
that file in my configuration contains directories like these:
/etc
/usr/bin

so you are going to do a command like this:

chmod -R a=rwx /etc
chmod -R a=rwx /usr/bin

YOU ARE GOING TO BREAK AND MESS UP YOUR ENTIRE SYSTEM!

Besides, why keep using an insanely insecure permission model like this ??
Good Luck

pierky
8 April 2013 at 16:45
Edited, thanks for your note!

Jignesh Shah
9 May 2013 at 08:55
I have installed FlowvIewer 4.0 and want to capture v9 packet from cisco asa 5500 device. Currently I have configured
V 5 and flow capture is started but I cant see any data in FLowViewer, FlowTracker and FlowGrapher. Can please let me
know the configuration of Flow-capture to capture v9 packets.

NetFlow: installation and configuration of NFDUMP and NfSen on Debian Pierky's Blog
16 July 2013 at 14:54
[...] the brief overview about the installation of flow-tools and FlowViewer, in this post Id like to share my experience
about the setup of a basic solution based on [...]

Jay
29 January 2014 at 02:32
Thanks Luca for your warning about the permissions. I just wish I had read all the comments BEFORE I did it. It did
indeed mess up my entire server but thankfully it was a VPS with snapshot backups so I was able to revert back to a
working state.
But I do agree, 777 on anything is a BAD IDEA.

Netadmin
4 July 2014 at 09:11
When I generate the repor, ntothing to see .
It say sh: 1: /usr/local/flow-tools/bin/flow-stat: not found.
How can I solve ?

Leave a Reply
Name *
Email *
Website

Type the text

Privacy & Terms

Post Comment (Ctrl + Enter)

Notify me of new posts by email.

Pierky's Blog.

Proudly Powered by WordPress | Nest Theme by YChong