Overcoming Common Causes for SIEM

Deployment Failures
in

Archived Published: 21 August 2014 ID: G00260858

Analyst(s): Oliver Rochford

Summary
Implementing SIEM continues to be fraught with difficulties, with failed and stalled
deployments common. Security managers can avoid the six most common failures by
following these best practices.

Overview
Key Challenges

 Security information and event management (SIEM) is a use-case-driven technology

and requires careful and meticulous planning to avoid stalled or failed deployments.
 Selecting a SIEM solution is predicated on a clear understanding of the scope,
objectives and associated use cases. Buying before planning can doom a deployment
before it has even begun.
 Many organizations underestimate the amount of prework that is required and hit a
hard stop once this becomes clear.
 The necessary resources and expertise for effective operations are routinely
misjudged, and SIEM is often selected even though the feasibility of running it in-
house was not assessed.

Recommendations

IT security planners and managers

 Evaluate requirements — including for ongoing operations — and follow a

formalized planning process before selecting a SIEM solution.
 Define the scope of the SIEM deployment and associated objectives.
 Plan for an initial six- to 12-month road map encompassing the deployment of the
SIEM solution and the phased implementation of five to seven use cases.
 Follow an output-driven SIEM model, and include only the data required to address
distinct use cases.
 Simulate use cases to verify their soundness and effectiveness, as well as to test the
incident response readiness of security personnel.
  Consider involving an external service provider if required by resource or expertise
constraints or strategy.

Table of Contents
 Introduction
 Analysis
o Undertake Careful Planning to Avoid Stalled or Failed SIEM Deployments
 Pitfall 1: Failure to Plan Before Buying
 Pitfall 2: Failure to Define Scope
 Pitfall 3. Overly Optimistic Scoping
 Pitfall 4: Monitoring Noise
 Pitfall 5: Lack of Sufficient Context
 Pitfall 6: Insufficient Resources
 Gartner Recommended Reading

Introduction
Due to their complexity and rigid requirements, SIEM projects often do not live up to users'
expectations, and failed or abandoned deployments are not uncommon. Moreover, Gartner
frequently speaks to clients who are purchasing their second or third SIEM solution after
finding that their incumbent solution does not meet their requirements. Lack of planning and
underappreciation of the operational and infrastructure requirements, as well as
underestimating the required resources and skills, are the most common root causes for failed
SIEM projects.

Even a successful SIEM deployment — whether the company acquired it or is running and
maintaining it — is an expensive and resource-intensive proposition. Approaching a SIEM
project without sufficient planning and without following a formalized process will make it
even costlier, with the risk of fewer benefits.

This research points out how to overcome the six most common pitfalls that Gartner
encounters as root causes for failed and stalled SIEM deployments. Avoiding these — and
following the best practices that accompany them — will help ensure an effective and
successful SIEM deployment.

Analysis
Undertake Careful Planning to Avoid Stalled or Failed SIEM Deployments

Pitfall 1: Failure to Plan Before Buying

Despite the common perception that SIEM solutions are complex and expensive, many
organizations buy their solution without following best practices and first evaluating and
scoping the project to determine that a given solution will meet all of their requirements. The
chance of successfully implementing a SIEM project without prior planning is slight; the
necessary investment in time, resources and potential additional costs will far outweigh the
perceived benefit of moving forward without planning. Gartner commonly encounters
organizations where a SIEM solution was acquired and has been quietly gathering dust ever
since.

The majority of SIEM solutions provide solid security information management (SIM) and
security event management SEM capabilities. However — in terms of available out-of-the-
box, third-party integrations — integrated workflows, auxiliary capabilities such as network
flow (NetFlow) and network monitoring, including, for example, data derived from deep
packet inspection (DPI) or NetFlow analytics (see "Use NetFlow Analysis to Gain More
Value From SIEM" ) will vary widely. More importantly, without sufficient planning, correct
scoping becomes guesswork.

Planning is also important to prove feasibility. In the course of planning, you may come to
the conclusion that the inherent requirements of SIEM mean that this is not a realistic option
for your organization. If that is the case, there are better options available — from using a
managed service to investing in something else first for lower effort and reduced cost-risk.

SIEM is not suitable for everyone. It requires expertise and dedicated resources, it relies on a
sane, well-formed operational environment, and it will not compensate for shortcomings in
investment, operational execution or skills.

Best Practice: Use a Formalized Planning Approach

Gartner recommends the following approach to planning and implementing a SIEM (see
"Planning for an SIEM Technology Deployment" ):

 Form a core project team whose primary responsibilities include defining the goals,
scope, and deployment phases of the project, and identifying initial stakeholders.
 Define monitoring objectives and the initial scope of deployment.
 Determine the use cases that will be covered.
 Define the data collection, retention, reporting and monitoring requirements.
 Assess how many and which data sources will be included to determine the required
scale of the solution — whether in terms of events per second, back-end storage or
processing power.

An organization should then use this foundation to:

 Conduct an environmental assessment.

 Assess the architectural requirements to determine what effort will be required to
integrate data sources, how many collector instances will be needed to accommodate
segmented networks or geographically dispersed locations, or whether other
organizational groups such as networking or application support should be included in
the planning.
 Define monitoring and incident response processes.
 Create the relevant processes and policies in order to assess how many resources will
be required to actually run the solution.
 Select the technology (see "Magic Quadrant for Security Information and Event
Management" and "Critical Capabilities for Security Information and Event
Management," as well as "Toolkit: Security Information and Event Management RFP
Toolkit ").
 Deploy the technology (see "How to Deploy SIEM Technology" ).
Pitfall 2: Failure to Define Scope

Attempting to deploy SIEM without a predefined scope can be sardonically expressed as: no
scope, no hope. The scope provides the basis for all that will follow — planning, deploying,
implementing and maturing the SIEM. It will determine the choice of solution, the
architectural requirements, the necessary staffing, and the processes and policies.
Deployment without a defined scope is like building a house without a foundation. Not only
is the process of building fraught with danger; the house will eventually crumble.

Attempting to define the scope afterward will be a costly venture. The technological debt
incurred — especially in terms of possibly having selected an unsuitable SIEM solution for a
scope retrospectively defined — may not be able to be paid. Gartner frequently speaks to
clients who have to write down their past investments in SIEM due to this particular problem.

Best Practice: Define Scope and Objectives Based on Drivers

SIEM scope and associated objectives will be determined by the drivers for deploying SIEM
and will facilitate the creation and design of use cases. Driverless SIEM will never provide a
return on investment.

Alongside a variety of less common niche drivers, there are two primary drivers for SIEM:
compliance and threat management (see "How to Implement SIEM Technology" ).

 Compliance — When compliance is a driver, the scope will be determined by the

compliance requirements and the regulatory mandates. Typical mandates include log
management or user activity monitoring by regular inspection of reports and
monitoring for policy violations. The main objective is to comply with regulations.

 Basic use cases include the collection and retention of log data, as specified by a
regulatory mandate, the occasional generation and review of user activity reports,
with advanced use cases ranging from the automated auditing of policy violations to
daily log reviews.

 Threat Management — Threat management scope can span many different

objectives and use cases, for example, detecting spear phishing and malware attacks,
or monitoring third-party or privileged users. Depending on the specific objective, the
scope will typically include network security devices — such as firewalls, intrusion
detection/prevention systems (IDS/IPS), Web application firewalls or proxies —
security applications like vulnerability assessment and data loss prevention (DLP), as
well as various security, audit and applications logs from systems or database servers.

 Some examples for threat management use cases are real-time monitoring for external
network threats, authentication failures, and user activity and behavior profiling and
monitoring. These will be accompanied by incident response processes and automated
cross-data correlation and alerting among others (see "Planning for an SIEM
Technology Deployment" — Note: Some of the preceding references link to
documents that have been archived; some content may not reflect current conditions).

 Niche — There are also a number of niche use cases with their own scope — for
example, fraud or suspicious transaction monitoring, or industrial control system
 monitoring. Fraud management is often focused on the application layer and
associated event sources and can expand the relevant stakeholders and involved
participants considerably.

Pitfall 3. Overly Optimistic Scoping

To obtain the maximum value out of a SIEM, it may seem tempting to do everything with it
at once. SIEM solutions can ingest and process vast amounts of event data and provide
capabilities to manage these accordingly. Critically though, the only effective way to scale
the SIEM to be the effective platform for organization-wide monitoring and incident
management is to do this in stages. Every use case has distinct required data sources and
subsequent correlation rules, alerts, reports and dashboards.

Attempting to throw everything at the SIEM at once and hoping to be able to clean up later is
one of the most common causes for stalled SIEM deployments. In many cases, an
environmental assessment of that scale is a daunting and costly task that inevitably involves
many different organizational units and stakeholders. Architectural and operational changes
that may result from the use-case requirements will be complex and difficult to implement
across the organization.

Best Practice: Construct an Initial Road Map of Five to Seven Use Cases

Gartner recommends that IT security managers planning to implement SIEM identify

between five and seven use cases that will be used to construct an initial road map. A realistic
time frame for this sort of implementation is six to 12 months, depending on use-case
complexity and depending too on whether additional technologies or stakeholders are
required. Then further use cases can be added going forward. Once the process has become
sufficiently formalized and practiced, many organizations find their ability to implement new
use cases becomes more efficient and effective.

It is however, not uncommon to have a combination of compliance, threat management or

niche drivers. In this situation, the use cases should be prioritized according to business
needs. Advanced users can also attempt to logically group use cases according to shared
requirements such as needing the same data sources.

Common use cases include authentication tracking and compromised account detection,
tracking compromised and infected systems, malware detection by using network
connectivity logs to identify outbound connections, and tracking system changes and other
administrative actions.

Pitfall 4: Monitoring Noise

SIEM is not log collection, where the goal is to capture and store all logs from all devices and
applications without discrimination. Yet a common mistake is to approach it this way,
thinking it will be easy to make sense of all of this data once it is in the SIEM. The
predictable result is that what should be an exercise in reducing noise actually amplifies it
and generates more of it. Finding a needle in the haystack does not benefit from increasing
the amount of hay.
In addition, beyond out-of-the-box correlation rules, reports and dashboards that cover
common basic use cases, SIEM has to be configured to look for and recognize the activity or
events you may be seeking. It is not a magic bullet — throwing data at it and hoping it will
automatically illuminate every security problem in your environment will yield nothing but
disappointment and disillusionment.

Certain anomaly detection approaches such as statistical analysis, deviation from baseline and
outlier identification can potentially benefit from larger datasets, but even then the specific
type of data is relevant. Despite some basic overlap in functionality and approach, SIEM is
not big data.

Best Practice: Employ Output-Driven SIEM

To facilitate the targeted and focused collection and analysis of only relevant events and data,
SIEM should be output-driven, which means that event data is captured only if it is required
for a predetermined output or result. Log and event sources must be admitted based only on
specific use cases, correlation rules, reports and dashboards. For example, for the typical use
case of monitoring for suspicious outbound connectivity and data transfers, firewall and Web
proxy logs and network flow data are required. The use case would not, however, benefit
from the inclusion of Web application access or Dynamic Host Configuration Protocol
(DHCP) logs.

Except for special use cases, the log collection and management infrastructure must be
implemented first. This will allow the controlled selective filtering and forwarding of relevant
and in-scope event data to the SIEM, permitting an output-driven approach. If there is a
requirement to gather all logs, for example, for later forensic usage or to fulfill nonsecurity
requirements, this can be done on the log collection and management tier without polluting
the SIEM.

Output-driven SIEM requires careful forward planning, know-how around real-world threats
(or regulatory requirements and controls), and expertise around the processes and
technologies to be monitored. Often this requires the involvement of stakeholders outside of
the security organizations (see "Using SIEM for Targeted Attack Detection" ). However, a
case can be made that these are general requirements for an effective SIEM deployment.

This investment in planning will yield a far more effective SIEM deployment compared to
doing this haphazardly. Output-driven SIEM has several benefits and advantages beyond
preventing the SIEM being inundated with useless data.

More focused use cases with selective data ingestion reduce the amount of personnel required
to watch and manage the SIEM and also minimize the required data collection. This allows
for more cost-effective scaling of the SIEM solution.

If you do not know what you are looking for, SIEM will not provide a lot of additional value
— especially for the potential costs involved.

Pitfall 5: Lack of Sufficient Context

Monitoring an intrusion detection system (IDS) via SIEM will add some powerful
capabilities: It will identify and group attacks originating from a repeated source, for
example, or it will apply threat intelligence to an otherwise anonymous IP address to indicate
whether that IP has been seen in other attacks. But without further context and data sources
for correlation, such as access to the application server logs to verify if the attack seen on the
IDS has been successful, this is as good as it will get. SIEM does not see anything that you do
not provide.

Best Practice: Follow a Formal Use-Case Implementation Process

Generally, these pitfalls can be avoided by following a formal process for use-case
implementation that includes the following:

 Use-case selection — Determine and select the initial use case(s), which identify
what you are trying to monitor or achieve.
 Data collection needed — Identify the scope of required data.
 Log source configuration needed — Configure necessary log and data sources to
send the data to, or fetch it from, the SIEM system.
 SIEM content creation, preparation and selection — Construct the SIEM content
(correlation rules, reports, dashboards, etc).
 Definition of operational processes required — Define the operational processes to
manage this specific use case if required.
 Test the use case — Generate the anticipated behavior or activity, and verify that
everything is configured correctly (see Note 1).
 Refine the content and processes loop — Remediate and refine any issues and
SIEM content.

Pitfall 6: Insufficient Resources

A successful SIEM deployment requires skilled people. Once you begin looking, you will
inevitably find things; and these findings will require a response and management.
Additionally, a SIEM solution does not run itself. At a bare minimum, it requires ongoing
tuning and maintenance to reflect changes in the environment, "threatscapes," compliance
mandates or the gathered data itself. There are three main duties associated with the operation
of SIEM (see "Security Information and Event Management Architecture and Operational
Processes" ):

 Run — This entails managing and maintaining the underlying infrastructure for the
SIEM, ensuring that patches have been applied, that there is sufficient storage or that
users are added or deleted. Typically, this is an engineering task, especially when
following segregation of duties per ITIL, for example.
 Watch — This encompasses real-time monitoring of alerts and events and
responding, investigating and escalating incidents. A typical role title would be
security analyst.
 Tune — This aspect focuses on the ongoing optimization and tuning of correlation
rules, reports and signatures, and even processes involved with the watch duties
described above. Often this is done by a senior security analyst or third line.

Each one of the duties requires time and a different skill set, and each has to be performed on
a continual basis.
While some airplane mechanics may be able to fly a plane, you would probably prefer a
fighter pilot to fly it into combat if the need ever arose. The same is the case with SIEM. The
maintenance and administration of the architecture remains under the engineering umbrella
and will feel familiar to anyone managing other types of systems or network infrastructure.
However, using SIEM — making sense of the data that is coming in, analyzing and
responding to incidents, and constructing use cases and correlations rules — requires a
security analyst. Most organizations will require several security analysts and will find that
they are in short supply 1 and come with a high price tag.

Not only must you have sufficient knowledgeable staff to manage and maintain the SIEM,
but you must also be prepared for the additional work that results from the SIEM. Incidents
must be investigated and issues remediated, and these tasks are seldom the responsibility of
the security organization; they require other departments and teams.

For a typical midsize bank, a minimum staff of eight is required to run a dedicated 24/7
security monitoring operation, with two analysts per shift (not necessarily dedicated full-time
equivalents) working three days and having four days off, reversing the week after. In total,
there are four 12-hour shifts per week, without taking into consideration vacation, sickness or
staff turnover. In addition, this does not include any managers, and it also does not account
for how much work is actually there. Rather, it is the minimum to allow real-time, 24/7
monitoring.

Best Practices: Limit the Scope and Engage an External Service Provider

SIEM is not a deploy-and-forget technology, and it does not run itself. Instead, SIEM is a
force multiplier. It will allow certain tasks and use cases to be done more efficiently and
effectively, but it will not run by itself. There are, however, a few strategies that can be
employed to operate SIEM with a low staff contingent:

 Limit the scope — Adapting the scale of what is being monitored to align with the
available resources is a viable option. This applies to the number of monitored data
sources, as well as the scope of the use cases. Use cases should be concise and
focused so they can be automated via correlation rules as much as possible, and there
should be a realistic estimate of how many use cases in total can be managed by the
available staff.

 Successfully implementing at least limited use cases can provide risk reduction where
it is most needed and can also be used to build a business case for expanding the
monitoring scope.

 Engage an external security service provider — Two very basic questions must be
answered in order to decide whether involving a service provider is a good solution:
1. Can I provide a better service or capability for the same or less cost than the
service provider? Can I provide it all?
2. Does the risk reduction I gain from delegating this to an external provider
outweigh the risks associated with handing over control?

If the answer to these two questions is affirmative, engaging a managed security services
provider (MSSP) will be a viable solution for many use cases, for example, log collection and
management for compliance, or external threat monitoring for threat management (see
"Choose Advanced Capabilities From Managed Security Services Providers With Care" ).

Fully resident staffing is also offered by many providers, but it can be prohibitively
expensive, with prices ranging from $100,000 to $350,000 per annum per analyst. 2

Additional research provided by Mark Nicolett, Kelly Kavanagh and Anton Chuvakin

Gartner Recommended Reading

Some documents may not be available as part of your current Gartner subscription.

"Planning for SIEM Deployment: Establish Scope and Requirements for Successful
Implementation"

"How to Deploy SIEM Technology"

"How to Implement SIEM Technology"

"Using SIEM for Targeted Attack Detection"

"Security Information and Event Management Architecture and Operational Processes"

Evidence
1
In "Critical Times Demand Critical Skills" (Frost and Sullivan in partnership with
Booz/Allen/Hamilton, 2013), the Security Analyst skill set was identified as representing the
greatest shortage, with 47% of the respondents naming that position as their top need.
2
Based on pricing reviewed by Gartner during client interviews.

Note 1
Testing
While testing may seem daunting to some, without it, it is impossible to know whether what
you intend on monitoring will actually be feasible. Similarly, simulated attacks must be used
to verify that sufficient forensic data is being gathered to allow effective incident or breach
investigation, especially sufficient to be used in legal proceedings.

External penetration tests prove an ideal testing bed to ensure that attacks are even detectable
with your current deployment. It is something that has to be verified before a breach occurs
— afterward it is too late.

© 2014 Gartner, Inc. and/or its Affiliates. All Rights Reserved. Reproduction and distribution of this publication
in any form without prior written permission is forbidden. The information contained herein has been obtained
from sources believed to be reliable. Gartner disclaims all warranties as to the accuracy, completeness or
adequacy of such information. Although Gartners research may discuss legal issues related to the information
technology business, Gartner does not provide legal advice or services and its research should not be construed
or used as such. Gartner shall have no liability for errors, omissions or inadequacies in the information contained
herein or for interpretations thereof. The opinions expressed herein are subject to change without notice.

Why Gartner

Gartner delivers the technology-related insight you need to make the right decisions, every
day.

Find out more

 Taha Khan |
 Log Off |
 Select a Gartner site

 Why Gartner
 Analysts
 Research
 Events
 Consulting
 About

 About
 Careers
 Contact
 Gartner Blog Network
 IT Glossary
 Newsroom
 Policies
 Privacy
 Site Index
 Webinars

© 2017 Gartner, Inc. and/or its Affiliates. All Rights Reserved.