Firewall Logging and Log Analysis

Dr. Anton Chuvakin 3/2010 This paper covers firewall logging, log management as well as use cases for firewall log management tools. It does not cover general purpose log management tools. If youd like to sponsor a modified version of this paper, please contact the author (anton@chuvakin.org)!

Contents
Introduction ................................................................................................................................. 1 Defining Features ....................................................................................................................... 2 Usage of Firewall Log Analysis Tools ......................................................................................... 3 Input data ................................................................................................................................ 3 Use Cases High Level ............................................................................................................. 4 Use Case Details .................................................................................................................... 5 Security ............................................................................................................................... 5 Network Management.......................................................................................................... 6 Firewall management .......................................................................................................... 7 Example Usage of Firewall Log Analysis Tool............................................................................. 7 Conclusions ................................................................................................................................ 7 One Thing ................................................................................................................................... 8 About author ............................................................................................................................... 9

Introduction
Enterprise network security is often visualized as a firewall protecting the internal network from malicious hackers outside. Indeed, firewalls from gigabit enterprise appliances to personal firewalls on each laptop or even mobile device are probably the most well known and also the most deployed component of information security. Apart from allowing and denying connections to and from the network, firewalls allow to record or log every single connection denied or allowed by the firewall. For example, connections from the outside in world to the DMZ web server or connections by users inside the company to their favorite social media web site.

Firewall logs typically look like this: Sep 07 2005 13:36:08: %PIX-2-106002: tcp connection denied by outbound list 1 src 10.10.110.186 4472 dest 10.10.83.189 57438 Or Sep 07 2005 13:36:08: %PIX-6-302001: Built outbound TCP connection 637515 for faddr 10.10.199.33/443 gaddr 11.11.15.161/3975 laddr 10.10.90.213/3975

Analysis of such logs is extremely useful for security, compliance, and even operational purposes such as network management, bandwidth management, etc. For example, on the compliance side, PCI DSS, HIPAA, NERC/FERC all have firewall logging implications. Firewall logs are also extremely useful for incident response and forensics since they can help identify the connectivity pattern and serve as poor man netflow. On top of this, firewall logs can be used to assess the health of the firewall itself and to optimize the ruleset performance. As a result, multiple tools have been developed just to analyze firewall logs and to extract value out of that data. In addition, general purpose log management tools are often used for firewall log analysis as well (for example , see Natural Flow of Log Management at http://chuvakin.blogspot.com/2007/01/natural-flow-of-log-management.html)

Defining Features
Lets look at what those firewall log analysis tools actually do under the hood. The following are some the defining features of most firewall log analyzer tools, from the freeware to enterprisegrade: 1. Collect firewall log data either from a live firewall or from log data archives: given that many firewalls log using syslog (stream via UDP port 514), some tools include an embedded syslog server to receive live syslog data. Others simply expect to find stored logs in some location or expect the user to import or upload log files. 2. Store firewall log data: since reporting requirements cover hours, days and sometimes weeks of log data, most tools include data store for firewall logs typically in the form of a lightweight database. Sometimes this database is used to only store reports and not the original logs themselves. This is where such tools deferred from general purpose log management tools that will typically store original records. 3. Reporting: this area is the key functionality of a firewall log analyzer log data is summarized and presented in the form of reports. Sometimes reports can also be emailed or otherwise delivered to data consumers. Reporting interface might also include minimal data enrichment functionality such as DNS resolution for IP addresses found in reports. 4. Alert: few of the tools can also alert operators based on conditions in the incoming data. For example, if firewall fail-over is triggered, the tool will issue an alert to an operator.

Tools that can receive live syslog streams from firewalls typically will have alerting features. 5. Ad hoc search: unlike general purpose log management tools, dedicated firewall analyzers rarely include full text search across all log data. Other components vary by tool; they might include an ability to pull firewall rules from the device for additional analysis.

Usage of Firewall Log Analysis Tools

In this section, well look at how firewall log analysis tools are actually used in practice.

Input data
As mentioned in the previous section, firewall analyzer tools consume firewall logs and produce hopefully- useful insight for their operators. There are two types of firewall logs and available for analysis:
y y

Traffic logs Non traffic logs

In particular, traffic logs include the following logs:

y

Allowed connections o Inbound o Outbound

Some firewalls would log two messages per each allowed network connection: the first when it is initiated and the second when it is terminated. The latter message often contains connection duration and the number of bytes transferred.
y

Denied connections o Outbound o Inbound

It should be noted that the separation into inbound and outbound logs is overly simplistic: most organizations operate firewalls with multiple segments and multiple network interfaces. At the very least, the company will have three interfaces for internal, external and DMZ where public network services are hosted. Thus firewall will log connections from inside to DMZ, from DMZ to inside (nothing should be allowed here so only denied connections should be logged), from inside to outside and from outside to inside (nothing should be allowed here as well), etc. Firewalls with many more networks zones and multiple interfaces are pretty common. Non traffic logs cover various firewall performance and administration messages, access to firewall system itself, as well as logs from other components of a multi-function firewall device.
y y

Access to firewall Changes to firewall o Rules

Configuration User accounts Issues with firewall

o o

A good example of a large set of such messages can be found here: System Log Messages http://www.cisco.com/en/US/docs/security/asa/asa72/system/message/logmsgs.html Overall, most log analysis tools focus on traffic logs. Non traffic logs are simply summarized and presented to the operator or, sometimes, even ignored.

Use Cases High Level

On a very high level, firewall log analysis tools are used so that their operator can answer the following questions in their corresponding domains of responsibility: 1. What is going on? 2. What is wrong? In more detail, most of the use cases fall under the following four categories: Security: this covers the use of the firewall for detecting and investigating attacks and other malicious activities. Some of the examples are:
o o o

Malware infections tracking via outbound traffic Unusually large file transfers Denial-of-service tracking

Regulatory compliance: this is related to information security and covers the use of firewall logs for tasks prescribed in regulatory documents such as PCI DSS. These typically include using the firewall for managing and monitoring connections to and from regulated environment, such as cardholder data environment (PCI DSS) or a network where patient health information is stored (HIPAA). The examples are:
o o

Connections to/from regulated environment Data transfers from regulated environment

Firewall operation: this use case category covers the use of firewall of data for tuning the firewall and ensuring its optimal performance. Some of the examples are:
o o

Rule performance tuning Firewall component resource utilization

Network management: this covers the use of firewall logs for troubleshooting network performance of the network that the firewall is connected to. These include bandwidth management as well as looking for misconfigured systems based on their network activity. The examples are:
o o

Misconfigured system detection Figure who is using all the bandwidth

Lets review some of the use cases in depth.

Use Case Details

Security Security usage of firewall logs analyzers includes looking for malicious hacking, malware and network access abuse. For example, a security analyst may run a few reports across log data from perimeter firewalls in order to determine which internal systems display infected-like behavior. Such behavior might include scanning internal or external systems through the firewall, repeated attempts to connect through the firewall to their command and control (botnet C&C) facility or attempts to perform denial of service attacks against third parties. These activities can be identified by looking at, for example:
y y y

Hosts with the highest number of connections to the outside world through the firewall Scanning internal hosts Known call-home malware traffic such as IRC or DNS-like traffic to non-DNS services.

Modern malware network traffic signatures are always evolving and there is no easy way to write rules for their identification. Identifying port scans directed at the company used to be at very common use for firewall logs 10 years ago. Today everybody knows that every single Internet exposed system is scanned all the time. It is however valuable to observe scans from the inside to the outside of the company (this needs to be done not out of caring for those targeted third parties, but as a means of identifying the infected systems in your organization). Still, firewall inbound logs can be used to detect probing in the form of host sweeps (one or few port hit on many systems) or port scans (many ports his on one or many systems), if such detection is desirable. Another emerging use for firewall logs is tracking network abuse and even insider attacks. These can also be called suspicious network usage.
y

y y

y y y

Largest file transfers outbound: reviewing a report that shows largest transfers of data from the internal network to the outside can be used to detect theft of intellectual property, regulated data or other valuable information. File uploads: looking as file uploads to social media, webmail or other public sites can be used to track information leakage. Suspected DoS tracking: infected machines are commonly used to send massive amounts of network traffic against the targets in order to stop their operation. Firewall logs can be used to detect such activity. Suspected spam sending: similarly, frequent connections to port 25 on external systems initiated by internal desktops is a sign of spam bot infection. New protocol usage: usage of an unusual protocol is a frequent sign of backdoors or cover channel communication of malicious nature New outbound port usage: just as new protocol usage, new outbound port usage is interesting from a security point of view.

Suspicious HTTP usage (commands, duration, etc): if firewall is capable of logging http you reptiles, it can be used for simple web access tracking. However, web proxies are better for this. Extra-long sessions: detecting and the sessions of unusual duration through the firewall can be used to detect bots and other malware connecting to its controllers as well as hacker attempts to communicate with internal machines.

Finally, security use for non traffic firewall logs is in monitoring command usage on the firewall as well as monitoring administrative access to the firewall. Another common security use for firewall logs is incident response. In this case, security analyst will search for particular IP address or track a particular connection across logs. They might also be search for protocol use tracking across suspected compromises. Please see my Logs for Incident Response presentation for more details (www.slideshare.net/anton_chuvakin). General purpose log management tools with advanced search capabilities are better suited for this type of usage, since they can go across multiple types of log data in at single search query. Network Management While firewalls originated as security devices, today they are commonly seen as parts of network infrastructure and often managed by the network, not security, team. They are used for productivity purposes such as bandwidth shaping as much or even more as for security. Here are some common uses full firewall logs in this domain. Network utilization tracking is a very common use for their logs. For example, tracking protocol usage for non business-related needs, such as:
y y y y y

Files transferred via FTP (uploads, downloads) Streaming media usage Non-business site usage (for firewalls capable of logging URLs) IM usage Inbound SSH, telnet, etc usage

Identifying users who access stream insights, webmail, instant messenger is often given to the firewall administrators as a task, especially at smaller companies. Other bandwidth management reports that may be used include:
y y y y y y

Network access abuse and waste Top network users Top network protocols/ports Top internal traffic destinations Top internal traffic sources Top websites used

As a result, firewall purchases are sometimes justified from the point of view of productivity and not from hacker protection.

Firewall management Using firewall logs for optimizing and managing the firewall itself is also very common; this is also the main use for non-traffic firewall logs. Common uses for log data include:
y y y y y y

Failed auth attempts to the firewall New firewall user accounts Unauthorized firewall configuration changes Firewall errors and failures Unauthorized firewall ruleset changes Firewall appliance component operation o VPN usage and operation o NIDS operation o Anti-malware operation Rule management o Rule performance: overused/underused rules o Never used rules o Newly added rules

Many of these do not require advanced analytics since the volume of such logs is significantly less than the volume of traffic logs. Simply displaying all records is frequently enough. Note that dedicated tools to perform firewall rule management and rule set analysis have emerged to help firewall administrators to tackle rapidly growing and unwieldy firewall rule sets sometimes numbering thousands of rules.

Example Usage of Firewall Log Analysis Tool

John, a firewall administrator and network manager at a small retail chain is in charge of operating firewalls protecting the data center. His tasks include updating and managing firewalls, fulfilling requests from business unit for network connectivity as well as monitoring for security issues - and many other IT tasks. Every few days, John starts his log analysis tool and runs a report that compares today traffic with yesterdays traffic pattern. That allows him to conclude that nothing unusual has taken place. He also reviews a report that shows administrative activity on the firewall to make sure that no user has logged in into his firewall. Next, he runs report that shows network connectivity across all protocols. This allows him to track for non-business use of network and then take actions against those who abuse network access privileges. Finally, he quickly checks the virus report from an embedded anti-virus module to check for virus activity on his network.

Conclusions
Firewall logs present one of the underutilized IT data sources. At the same time,

firewall log data is extremely useful for security, regulatory compliance and network operations purposes. To handle large volume of firewall log data (often going into gigabytes a day) requires using either a dedicated firewall log analysis tool or a scalable enterprise log management tool. Firewall log analysis usage by firewall administrators, network engineers, security analysts and incident responders includes both looking at the big picture of network traffic across the organization as well as looking for suspicious and malicious activities. If your organization isnt paying attention to this type of data, now might be at good time to start.

One Thing
If, after reading this paper, you decided to take action, but you only have time to do one simple thing (everybody knows how busy IT and IT security professionals are), then consider logging all connections from inside your network to the Internet (outbound connection) and analyzing that data. Egress filtering and outbound logging have been among recommended security practices for many years, but they still have not become popular despite their huge utility. Doing this might save your behind not only from attackers, but also from auditors or angry users in your organization.

Key Firewall Logging Resources

This section contains a few resources, related to firewall logging, log analysis and log management: y y Simple log review checklist, including firewall data: http://chuvakin.blogspot.com/2010/03/simple-log-review-checklist-released.html SANS log management class (covers firewall logging): http://www.sans.org/security-training/log-management-in-depth-compliancesecurity-forensics-troubleshooting-2912-tid Open source log management tools, including tools useful for firewall log data analysis: http://securitywarriorconsulting.com/logtools/ Firewall log analysis primer: http://www.secureworks.com/research/articles/firewall-primer Cisco ASA and PIX Firewall Logging: http://www.ciscopress.com/articles/article.asp?p=424447&seqNum=4 Check Point Firewall Log Analysis In-Depth: http://www.sans.org/reading_room/whitepapers/logging/check-point-firewall-loganalysis-in-depth_33228

y y y y

(Old) Firewall Logging & Monitoring http://www.loganalysis.org/sections/parsing/application-specific/firewalllogging.html

About Author
Dr. Anton Chuvakin (http://www.chuvakin.org) is a recognized security expert in the field of log management and PCI DSS compliance. He is an author of books "Security Warrior" and "PCI Compliance" and a contributor to "Know Your Enemy II", "Information Security Management Handbook" and others. Anton has published dozens of papers on log management, correlation, data analysis, PCI DSS, security management (see list www.info-secure.org) . His blog http://www.securitywarrior.org is one of the most popular in the industry. In addition, Anton teaches classes and presents at many security conferences across the world; he recently addressed audiences in United States, UK, Singapore, Spain, Russia and other countries. He works on emerging security standards and serves on the advisory boards of several security start-ups. Currently, Anton is developing his security consulting practice www.securitywarriorconsulting.com, focusing on logging and PCI DSS compliance for security vendors and Fortune 500 organizations. Dr. Anton Chuvakin was formerly a Director of PCI Compliance Solutions at Qualys. Previously, Anton worked at LogLogic as a Chief Logging Evangelist, tasked with educating the world about the importance of logging for security, compliance and operations. Before LogLogic, Anton was employed by a security vendor in a strategic tool management role. Anton earned his Ph.D. degree from Stony Brook University.