McAfee Enterprise Security Manager

Writing Custom Parsing Rules in McAfee ESM

September 19, 2016

The information contained in this document is confidential and proprietary.

Please do not redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 1 of 21

Table of Contents
1 Introduction .................................................................................................................................................................................3
2 Prerequisites................................................................................................................................................................................3
3 Creating a Custom ASP Rule ................................................................................................................................................4
3.1 Create a New Data Source ...........................................................................................................................................4
3.2 Create a Custom Parsing Rule for ASP ..................................................................................................................5
3.3 Enable the New ASP Rule in Policy ...................................................................................................................... 15
4 Creating a New Custom Type ............................................................................................................................................ 16
4.1 Create a Custom Type ................................................................................................................................................ 16
5 Testing the New ASP Rule .................................................................................................................................................. 18
5.1 Importing a Log Sample ............................................................................................................................................ 18
6 Best Practices ........................................................................................................................................................................... 20
6.1 Know Your Data ............................................................................................................................................................ 20
6.2 Regular Expressions .................................................................................................................................................... 20
6.3 Content ............................................................................................................................................................................. 21

The information contained in this document is confidential and proprietary.

Please do not redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 2 of 21

1 Introduction
This guide details how to create and deploy a custom advanced syslog parser (ASP) rule in the
McAfee ESM. A custom ASP rule could be necessary if there is an unknown/unparsed log in the
McAfee ESM. This guide also includes some Best Practices to follow when writing ASP rules.

2 Prerequisites
The following will be required, or useful, in creating custom parsing rules:
A sample of the log to parse.
Account privileges to add data sources. In some cases, administrator privileges, such as
the ability to define custom types may also be needed.
Vendor documentation can be helpful to determine the meaning of each field in the
logs desired to be parsed.
A working knowledge of PCRE (Perl Compatible Regular Expressions).

The information contained in this document is confidential and proprietary.

Please do not redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 3 of 21

3 Creating a Custom ASP Rule
Follow the steps below to build a new SYSLOG data source and create an ASP parsing rule.

3.1 Create a New Data Source

This step is optional if a data source already exists and is collecting the logs needing to be
parsed.
1. Select the Receiver, and then click on the + icon to add a data source.

2. The Add Data Source window will open, set the Data Source Vendor as Generic and the
Data Source Model as Advanced Syslog Parser. Give the new data source a name, add
the proper IP address, and ensure that the Support Generic Syslogs field is left as Do
nothing.

3. Select OK, and then apply the new data source settings by confirming in the next
window. This will create a new data source with an empty rule policy.

The information contained in this document is confidential and proprietary.

Please do not redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 4 of 21

 3.2 Create a Custom Parsing Rule for ASP
1. From the McAfee ESM console, select the data source and open the Policy Editor.

2. The Policy Editor window should appear. In the menu on the left, expand Receiver by
selecting the + symbol, and select Advanced Syslog Parser. Initially this list will be
blank while looking at the policy for the new data source.

3. Select New > Advanced Syslog Parser Rule.

The information contained in this document is confidential and proprietary.

Please do not redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 5 of 21

 4. The Advanced Syslog Parser Rule window will appear. Within the General tab, assign
information as displayed below:

Name Assign a name, making it as descriptive and unique as possible. This is the
event message that will show in the ESM views when the rule matches a log, unless the
message is mapped directly from the log text within the rule.
Tags Assign tags to the rule. It is best practice to assign one or more tags to which this
rule belongs. This helps in finding and grouping sets of rules created for a given device
or application within the policy editor. Any tags added to a rule, will cause ESM to
automatically include the rule in any policy that has enabled the given tagged rule set.
Default Normalized ID The normalization or category is a key step as many views,
correlation rules, and reports use this field as a filter. It is very important to select the
most relevant value in the taxonomy for the parsing rule in order to get the greatest
value from within the McAfee ESM.
Default Severity If the log message does not contain a value for the severity, then the
event will be assigned the value that is set here (the default is 25, with a scale of 1-100,
1 being the lowest and 100 being the highest).
Rule Assignment Type Rules can be grouped together; this pull-down menu provides
a list of supported products to group the parsing rules by, separating the events from
other data sources. This allows the event to be reported for a specific product.
Description Give the new rule a clear human-readable description. It should be a clear
and complete description of the rule with complementary information to what was
given in the rules name.

The information contained in this document is confidential and proprietary.

Please do not redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 6 of 21

 5. Click on the next tab, Parsing, assign information as displayed below:

Process Name This is an optional filter that acts similar to the content string, but only
applies to the process name found within the SYSLOG header. Due to various syslog
header formats used, and no guarantee that a syslog header will be present in the log, it
is recommended to use content strings instead of the process name match.
Content String If a fixed string is always going to be found in the log then add it as a
content string. The content string(s) of an ASP rule should uniquely identify each log if
possible. Content strings are very important as they will not only make matches more
accurate, but will speed up the execution of the rules in the running policy. It is highly
recommended to have one or more content strings per ASP rule. This is a pre-filter for
optimization only logs that match the given content strings will be considered for
matching and parsing by the regular expressions. The log must contain all given content
strings, otherwise the logs will not be parsed using the regular expressions.
Regular Expression The first regular expression determines if the ASP rule will match
the log. Any additional expressions will be used to capture values from the log that may
or may not be found in all logs.
Named Captures Named captures can be used in regular expressions to more easily
identify capture groups. The label used for the named capture can consist of alpha-
numeric and underscore characters but cannot begin with a number or include a space.
The regular expression syntax for a named capture is: (?P<NAME>regular expression
capture). Here is an example of a named capture where hostname is the name
assigned to the capture group: Host\x3d(?P<hostname>\S+). When using named
captures the policy editor will display the capture name instead of the capture number,
in the right-hand side of the Parsing tab as shown below.
The information contained in this document is confidential and proprietary.
Please do not redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 7 of 21

 Targeted Regular Expressions Targeted regular expressions allow for applying an
additional regular expression to the resulting value from another capture. For example,
a regular expression could be written to pull a users full name from a log. Referencing
that capture as the target, two targeted regular expressions could then be written to
pull out the users first and last name. This data can then be mapped to different
database fields. Follow the steps below to create a targeted regular expression:

1. On the Parsing tab, add a regular expression.

2. Add another regular expression and select the capture (name or number) from
the Target key drop-down list that references the first expression. The second
regular expression should contain an expression to capture additional
information from the captured value of the first regular expression.

In the image below, the named capture group full_name has a value of Sam Johnson.
This named capture is being used as the Target Key for the first_name and last_name
targeted regular expressions. The regular expression for first_name and last_name will
be evaluated against the value of the full_name named capture. The result shows the
key of first_name:1 with a value of Sam and last_name:1 with a value of Johnson.
This specific example could be done within a single regular expression, and is a
simplified example to illustrate the purpose of the Target Key field used with regular
expressions. In general, it is recommended to run as few regular expressions as
possible, but that may vary with specific use cases.

The information contained in this document is confidential and proprietary.

Please do not redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 8 of 21

 Sample Log Data In this field a sample log can be added and will be saved with the
rule. The parts of the log matching defined regular expressions will be highlighted blue.
Format ASP can pre-process certain logging formats to simplify the mapping of data.
The following formats are available:
Generic This is the default and should be used if the log does not match the
other available formats.
CEF (Common Event Format) This eliminates the need to create a regular
expression for each capture, and will allow the data to be mapped using the
CEF key names found in the log.
JSON Similar to CEF, this eliminates the need to create a regular expression
for each capture, and will allow data to be mapped using the JSON key names
found in the log.
XML - Basic, Simple, or Positional This will allow ASP to parse logs that are in
XML format and assign parsed data. The XML format choice will depend upon
the type of XML that is within the logs.
XML Basic expects XML without any repeated elements.
XML Simple expects XML with either a single node with attributes, or
a single set of non-repeated elements without nesting.
XML Positional expects XML that can have multiple nodes with
attributes and multiple repeated elements with nesting.

The information contained in this document is confidential and proprietary.

Please do not redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 9 of 21

 Parsed Values The Key/Value fields on the right displays what is being parsed from
the log sample(s) by the regular expression(s). The Key will display two numbers,
divided by a colon. The first number indicates the regular expression being used, and
the second number indicates the capture group used within that regular expression. If a
captured value is the 4th capture in the 3rd regular expression defined, the key would
display 3:4.
6. Add applicable Content Strings by selecting Edit. These are very important and can
drastically affect performance of the created rule. Any content strings added will need
to be a string that is guaranteed to be within the log, as these act as a type of pre-
filter. Multiple unique content strings can be added to increase the speed and
efficiency of the rule.
7. Select applicable regular expression options by adding checks within the boxes add one
or more Regular Expressions by selecting the + symbol.

Only use regular expressions for parsing purposes Checking this box will remove the
requirement for the first regular expression to match the log. The parser will rely upon
the Content String for matching purposes, and will only use regular expressions to
parse data when it finds a matching pattern.
Case Insensitive If the log may contain either upper or lower case letters in some
fields, it may be simpler to write the expression in the same case and then use this
option. This enables the case insensitivity option for all regular expressions defined in
the parsing rule.
Trigger when data doesnt match This option will make the rule trigger when the
regular expression does not match the log.
a. The Edit Regular Expression window will appear. Add the expression and
select OK.

8. With the expression added, the parsed values should be highlighted in blue, with the
Group and Value shown to the right. Verify that each value is parsed out correctly and
that all of the log line samples match. Make sure to know the meaning (from the vendor
device documentation or own knowledge) of all the fields within the logs and capture
the values that are useful.
The information contained in this document is confidential and proprietary.
Please do not redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 10 of 21

 9. After all the regular expressions are written to extract out the needed values from the
log, select the Field Assignment tab at the top of the window.
10. Available fields will be displayed, along with the Group and Value section as shown in
the previous Parsing tab.

11. Drag and drop the Values from the right hand side to the Expression column next to
the Field on the left.
a. For more complicated log file formats, multiple regular expressions may be
used to capture the same data. In such cases, a field such as First Time might
come from one of several potential several expressions. McAfee ESM allows the
ability to specify the regular expression values in order of preference, so that if
one does not match, it will use the second, or third, or so on. An example of this
is shown below:

12. If the field is not displayed that is needed, click + above the Sample Value column, to
display all custom type fields.

13. The Custom Types window will now be displayed. Select the desired field and then
select OK.

The information contained in this document is confidential and proprietary.

Please do not redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 11 of 21

 14. If a custom type does not exist for the type of data being mapped, the McAfee ESM
allows for user-defined custom types to be added. Once a new custom type has been
created, it will be listed within the Custom Types window and be available for use to
map the values to. Please see Section 4 for details on creating a new custom type.
15. Click on the final tab, Mapping. This section will allow parsing the date format, mapping
the action, and mapping the severity found within the log message.

The information contained in this document is confidential and proprietary.

Please do not redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 12 of 21

 Time Format The date/timestamp of a log message can be parsed using the variables
defined in these fields. Many standard date/timestamps are recognized automatically
by the McAfee ESM, however there may be a format that is either not recognized or that
would be preferred to display differently. This section will allow for formatting the time
to show up in the proper format when parsed.
Action Mapping Use this option if there is an action found within the log to be
mapped to an available McAfee ESM action.
Severity Mapping The severity mapping allows for a value within the log to be
mapped to a severity from 1-100. For example, a vendor might define their severity as
either Low, Medium, or High in their logs. With the Severity Map section the
severity value can map Low as 25, Medium as 50, and High as 75.
16. To add a Time Format, select the + symbol in the upper-right of the time format
section.

a. The Add Time Format window will appear.

b. Enter a custom time format. As the values are entered, a small window will
appear to assist in creating the custom time. The values can either be typed or
selected from the given list.
c. Select the fields that this custom format will apply to. It is best practice to apply
it to both First Time and Last Time.
d. Select OK to save.
17. To add an Action Map, find an applicable Action Value by scrolling through the list and
then click inside the Action Key column. Enter the value that will be parsed from the
device logs. If this value is parsed and assigned to Action, then it will map to the
assigned Action Value.

The information contained in this document is confidential and proprietary.

Please do not redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 13 of 21

 Below the Action Mapping table, the option will be given to use a default action when
one is not specified. This is generally a good option to set in order to insure that all logs
receive an action type. Depending on the device, the default action should be different.
To add the default action map, place a check next to Use the following action for the
default if one is not specified, and then select a default action from the drop down
menu.

18. To add a Severity Mapping, select the + symbol in the upper-right of the Severity
Mapping section.

a. The Add Severity windows will appear.

b. Within the Severity Mapping field, enter the value that will be parsed from the
device logs. If this value is parsed and assigned to Severity, then it will map to
the assigned Severity Value.
c. Move the sliding scale to the desired Severity Value.
d. Select OK to save.
19. Below the Severity Mapping section, the option to add a default severity will be given.
This is generally a good option to set in order to insure that all logs receive a severity
value. Depending on the device, the default severity should be different. To add the
default severity, place a check next to Use the following severity for the default if one
is not specified, and then enter a default value.

20. With the previous steps complete, the new ASP rule should be finished. Select Finish,
located in the bottom-right hand corner to finish and save.

The information contained in this document is confidential and proprietary.

Please do not redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 14 of 21

 3.3 Enable the New ASP Rule in Policy
Once the new rule has been created, it will need to be enabled and added to the policy in order
for the receiver to begin parsing the associated logs.
1. Open the Policy Editor for the data source that the new ASP rule was created for, and
beneath Receiver, select Advanced Syslog Parser. The newly created ASP rule(s) will
be listed with the Action column displaying disabled. Click on disabled and then
change the value to enabled.

2. With the new rule set as enabled, click on the Rollout icon displayed in the upper-right
hand corner of the Policy Editor window.

3. If the rule didnt already save previously, it will prompt to save. Select Yes to save.

4. Once saved, the Rollout window will appear and will display a list of devices to roll
policy out to. Ensure that the applicable data source is set to Roll this policy out now.
and select OK to do so.

The information contained in this document is confidential and proprietary.

Please do not redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 15 of 21

4 Creating a New Custom Type
4.1 Create a Custom Type
To create a custom type, login with a user that has system administrator privileges and then
select the System Properties icon.

1. Select the Custom Types option from the menu on the left.

2. Select Add from the options shown on the right.

Note: Select the Edit option to edit a user defined pre-existing custom type if
needed.
3. The Add Custom Type window will appear. Setup the new custom type as desired and
select OK.

The information contained in this document is confidential and proprietary.

Please do not redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 16 of 21

 Name This will be the name for the new Custom Type that will be displayed within
McAfee ESM.
Data Type This will be the type of data that the McAfee ESM will expect to be
populated into this field. Selection of the correct data type is important. Commonly
used data types include: Random String, IP Address, Decimal, Integer, and GUID.
Events Field These fields are for Events. There are a specific number of fields that can
be used to have data parsed into. If two different custom types occupy the same field,
only one can be used. When creating custom types, make sure to place it within a field
that is not already being used by another important field that will be used.
Flows Field These fields are for Flows. There are a specific number of fields that can
be used to have data parsed into. If two different custom types occupy the same field,
only one can be used. When creating custom types, make sure to place it within a field
that is not already being used by another important field that will be used.
Index Data This option will determine on whether or not the data that is parsed is
indexed within the database. Indexed data is searchable within the McAfee ESM.
Description This will be a simple description to explain what type of data the newly
created custom type contains.
4. With the desired custom type setting entered, select OK.

The information contained in this document is confidential and proprietary.

Please do not redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 17 of 21

5 Testing the New ASP Rule
5.1 Importing a Log Sample
Once the new ASP rule has been created and added to the current policy, it is a good idea to
test it to ensure that it is parsing as expected.
1. Obtain sample logs from the device that the new rule was created for and save these to
a text file.
2. Select the data source to be tested and click Properties.

3. The Data Source Properties window will appear, select the Upload button.

4. Navigate to the log sample file and select it to upload. Select the Upload button and
then a confirmation window will be displayed showing the file was successfully
uploaded. Select Close to close out the window.

The information contained in this document is confidential and proprietary.

Please do not redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 18 of 21

 5. To initiate the log and flow gathering process manually, click on the Get Events and
Flows button.

6. The Get Events and Flows window will appear. Select Events and then click the Start
button to collect the events.

7. Once the events are collected, find the events in the dashboard and verify the newly
created ASP rule is parsing as expected. If any adjustments need to be made, make the
changes and then send the sample logs through again to verify that the rule is parsing
as expected.

The information contained in this document is confidential and proprietary.

Please do not redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 19 of 21

6 Best Practices
The following are given as best practices that can help with creating ASP rules that work well
within the McAfee ESM.
Following these recommendations will help ensure reporting is accurate, not overly complex,
and will help parsers and Receivers maintain intended performance and efficiency.

6.1 Know Your Data

It is important to understand the logs that are being sent to the SIEM, and whether or not they
have value in your organizations reporting needs. It is recommended to only parse logs that
meet your organizations reporting goals.
Ensure that captured values mapped to ESM database fields align with the intended use
of the specific custom type fields.
Avoid indexing fields that contain unique and random or high cardinality data such as
URLs.
Ensure rules mapping event messages directly from the log are not mapping unique
and random or high cardinality strings as messages. The ESM creates a data source rule
for each unique event message, and ESM performance can be negatively impacted if
there are a high rate of unique strings being mapped to message.
Categorize events by adding a normalized category to the rule. Data source rules,
generated by parsing rules, inherit the normalization assigned to the main parsing rule.
This means that if the main parsing rule is left normalized to Uncategorized, then the
parsed events will also be normalized as Uncategorized, making a search for
Uncategorized events to find unparsed events in accurate.

6.2 Regular Expressions

By default, the first regular expression determines if the ASP rule will match the log and will
need to match a pattern that is always present in the log. Additional regular expressions can be
written to capture values from the log and mapped to custom types within the McAfee ESM.
Subsequent regular expressions do not determine the rule match, and are used for parsing
only.
While it is possible to test regular expression results on a few log lines within the McAfee ESM
console itself, we recommend using a graphical tool. There are many free web based tools that
can be used in addition to standalone installable tools. Optionally, another useful tool would be
a text editor that supports regular expression searches. Any tools used to test regular
expressions need to support pcre expressions.
Make sure that regular expressions are efficiently written because poorly written expressions
can take more system resources to run, and can affect the parsing performance of a specific
data source or even the entire Receiver.

The information contained in this document is confidential and proprietary.

Please do not redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 20 of 21

 6.3 Content
Ensure there is at least one value in the content field section. Content strings should be at least
3 characters in length and should be as unique as possible for the specific event. It is advised to
include enough content matches to uniquely identify the log. Using one or more content fields
within the ASP rule will significantly speed up the matching and parsing process on the
Receiver.

The table below shows an example of how to use the content field to uniquely identify a log
sample.
Syslog Sample Content Fields

<180>Jan 1 00:00:00 testhost ftpd[4325]: FTP ftpd

LOGIN FROM test.org [192.168.1.1], FTP LOGIN FROM
anonymous
<180>Jan 1 00:00:00 testhost ftpd[4325]: FTP ftpd
LOGIN REFUSED (ftp not in /etc/passwd) FROM FTP LOGIN REFUSED
192.168.1.1 [192.168.1.1], anonymous

The information contained in this document is confidential and proprietary.

Please do not redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 21 of 21